Data Skeptic Networks for AB Testing
6 snips
Nov 25, 2024 Wentao Su, a data scientist at ByteDance, specializes in A/B testing for social media platforms. He dives into the challenges of A/B testing in dynamic networks, highlighting the spillover effects that can distort results. Wentao introduces innovative strategies like one-degree label propagation to optimize test accuracy. He also explores how user interconnectedness impacts both experimental design and user experience. Finally, he discusses the significance of robust data processing techniques to effectively manage large-scale experiments.
AI Snips
Chapters
Transcript
Episode notes
A-B Testing Challenges in Social Networks
- A-B testing in social networks presents unique challenges due to interconnectedness.
- User interactions and influence can contaminate control groups, skewing results.
Geographic A-B Testing Challenges
- Kyle Polich faced similar A-B testing challenges in local search with regional advertisers.
- They mitigated spillover by dividing the U.S. into four zones and minimizing overlap.
Spillover Effects in Social Networks
- Social networks complicate A-B tests because viewers interact with creators and each other.
- Spillover effects arise as viewers in different groups influence each other, leading to underestimation of true impact.