Experiment Nation: The Podcast
S3E21 - A/B Testing Statistics Concepts Experimenters must know with Ronny Kohavi
Jun 4, 2023
Data scientist Ronny Kohavi discusses A/B testing statistics concepts all experimenters must know, including the Overall Evaluation Criterion, Twyman's law, and the amount of traffic needed for A/B tests. He shares insights on handling tests that turn out to be wrong and presenting experiment results to the CEO of Amazon. The podcast also highlights the benefits of AV tests and variety in designs for experimentation and CRO, while cautioning against the pitfalls of long-term testing.
52:29
Episode guests
AI Summary
Highlights
AI Chapters
Episode notes
Podcast summary created with Snipd AI
Quick takeaways
- Metrics become less reliable in long-term experiments, highlighting the need for careful evaluation.
- Using qualitative testing methods can provide insights for low-traffic websites until larger traffic volumes are reached.
Deep dives
The Importance of Running Experiments
The podcast episode emphasizes the significance of running experiments and gathering data over a long period. It highlights that users who continue using a service for an extended period differ from other users in multiple ways, which could introduce bias. It also warns that metrics like revenue per user could become less reliable when running experiments for a prolonged time. This underlines the need to carefully consider the reliability of metrics when conducting experiments and to avoid overreliance on long-term experiments.
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
