Lenny's Podcast: Product | Growth | Career

The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

Jul 27, 2023

01:23:07

Snipd AI

Renowned expert Ronny Kohavi discusses fostering a culture of experimentation, surprising A/B test results, the impact of small changes in search results, striking a balance between revenue and user experience, using data and case studies to convince teams, the 80% rule and the value of running experiments, avoiding sample ratio mismatches, favorite books and personal experience with Chernobyl, and the importance of control experiments.

AI Summary

Highlights

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Experimentation is crucial in driving product decisions and should be embraced even for high-risk, high-reward ideas.

Building an experiment-driven culture and prioritizing trust are essential for successful experimentation.

Small changes can have significant impacts, and incremental improvements should be pursued despite a high failure rate in experiments.

Deep dives

The Importance of Testing Everything

The speaker highlights the significance of conducting experiments to test any code change or feature introduction. Even small bug fixes or changes can have unexpected impacts. Experimentation is emphasized as a crucial practice, and high-risk, high-reward ideas should be pursued. The speaker advises being prepared for failure and emphasizes the value of testing big ideas, despite the likelihood of a high failure rate.

Bugs in Experiments

00:46

When Should Startups Start Running A/B Tests?

00:36

Small Steps with A/B Testing Trump Big Redesign Every Time

01:15

The Importance of Investigating Results and Tuyman's Law

01:42

Introduction

4min

Unexpected A/B Test Results: Boosting Revenue Without Hurting the User Experience

4min

The Impact of Small Changes in Search Results

20min

Striking a Balance: Importance of Overall Evaluation Criterion in A/B Testing

13min

Using Data and Case Studies to Convince Teams

2min

The 80% Rule and the Value of Running Experiments

12min

Avoiding Sample Ratio Mismatches in A/B Testing

19min

Favorite Books, Movie/TV Show Recommendations, and Personal Experience with Chernobyl

7min

Conclusion and Recommendations

2min

Brought to you by Mixpanel—Event analytics that everyone can trust, use, and afford | Round—The private network built by tech leaders for tech leaders | Eppo—Run reliable, impactful experiments

—

Ronny Kohavi, PhD, is a consultant, teacher, and leading expert on the art and science of A/B testing. Previously, Ronny was Vice President and Technical Fellow at Airbnb, Technical Fellow and corporate VP at Microsoft (where he led the Experimentation Platform team), and Director of Data Mining and Personalization at Amazon. He was also honored with a lifetime achievement award by the Experimentation Culture Awards in September 2020 and teaches a popular course on experimentation on Maven. In today’s podcast, we discuss:

• How to foster a culture of experimentation

• How to avoid common pitfalls and misconceptions when running experiments

• His most surprising experiment results

• The critical role of trust in running successful experiments

• When not to A/B test something

• Best practices for helping your tests run faster

• The future of experimentation

—

Enroll in Ronny’s Maven class: Accelerating Innovation with A/B Testing at https://bit.ly/ABClassLenny. Promo code “LENNYAB” will give $500 off the class for the first 10 people to use it.

—

Find the full transcript at: https://www.lennysnewsletter.com/p/the-ultimate-guide-to-ab-testing

—

Where to find Ronny Kohavi:

• Twitter: https://twitter.com/ronnyk

• LinkedIn: https://www.linkedin.com/in/ronnyk/

• Website: http://ai.stanford.edu/~ronnyk/

—

Where to find Lenny:

• Newsletter: https://www.lennysnewsletter.com

• Twitter: https://twitter.com/lennysan

• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/