High Signal: Data Science | Career | AI Episode 4: How to Build an Experimentation Machine and Where Most Go Wrong
12 snips
Nov 7, 2024 Ramesh Johari, a Professor at Stanford University, dives into the evolution of online experimentation, especially for tech companies and marketplaces. He discusses how organizations can shift to self-learning models and the common pitfalls they encounter, such as risk aversion. The conversation touches on the transformative impact of generative AI on experimentation processes. Ramesh also shares strategies for cultivating a culture of learning from failure and integrating data scientists to enhance business value, all while moving beyond traditional A/B testing methods.
AI Snips
Chapters
Transcript
Episode notes
A/B Testing as Standard Practice
- If you're a business leader, A/B testing should be standard practice when trying new things.
- The question is, where do we go from here, what's working, what's not working, and how do we move forward?
Evolution of Experimentation
- Organizations evolve in their experimentation sophistication, starting with basic A/B testing to pick winners and losers.
- More advanced organizations focus on 'why' an experiment worked, not just 'what' worked.
Hypotheses in Experimentation
- Ramesh Johari points out that classic science experiments start with hypotheses, unlike many tech A/B tests.
- He uses the example of boiling water to illustrate how hypotheses drive scientific inquiry.