Adventures in Machine Learning A/B Testing with ML ft. Michael Berk - ML 181
Jan 2, 2025
Michael Berk, a data scientist at Tubi, specializes in A/B testing and machine learning for streaming services. He dives into how A/B testing proves causality, emphasizing its crucial role in data-driven decisions for businesses. The discussion contrasts frequentist and Bayesian methodologies, highlighting sample size importance. Berk also shares insights on adapting to varying business environments and the shift from viewer count to watch time metrics in streaming. He wraps up with thoughts on the need for clear communication of data science principles to engage interest in machine learning.
AI Snips
Chapters
Transcript
Episode notes
Sample Size Calculation
- Use a sample size calculator to determine the necessary users for an A/B test.
- This calculator considers type 1 and 2 error rates and minimum detectable effect.
Variance Reduction
- Reduce variance in A/B testing by forecasting it with pre-experiment data.
- This isolates the signal from the noise, allowing for smaller sample sizes.
Minimum Detectable Effect
- Let the business define the minimum detectable effect for A/B tests.
- They understand the required lift for new features to be worthwhile.