Data Skeptic cover image

Data Skeptic

Networks for AB Testing

Nov 25, 2024
Wentao Su, a data scientist at ByteDance, specializes in A/B testing for social media platforms. He dives into the challenges of A/B testing in dynamic networks, highlighting the spillover effects that can distort results. Wentao introduces innovative strategies like one-degree label propagation to optimize test accuracy. He also explores how user interconnectedness impacts both experimental design and user experience. Finally, he discusses the significance of robust data processing techniques to effectively manage large-scale experiments.
36:42

Episode guests

Podcast summary created with Snipd AI

Quick takeaways

  • Network science enhances A-B testing by addressing spillover effects from user interactions, ensuring more reliable experimental outcomes in social media.
  • Temporal dynamics of different networks affect A-B testing results, necessitating careful selection of time frames for accurate data collection and analysis.

Deep dives

Challenges of A-B Testing in Connected Networks

A-B testing within social networks presents unique challenges due to the interconnected nature of users. Traditional A-B testing relies on creating distinct groups that are isolated from each other, but in a social network, the overlap between test and control groups can lead to influence. For example, if one group's engagement affects the behavior of another due to shared social ties, the integrity of the A-B test may be compromised. This issue necessitates innovative methodologies, such as clustering approaches, to ensure more accurate measurement and analysis in such dynamic environments.

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.
App store bannerPlay store banner

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode