Lenny's Podcast: Product | Growth | Career

The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

Jul 27, 2023

Ronny Kohavi, a leading expert in A/B testing with a rich history at Airbnb, Microsoft, and Amazon, shares invaluable insights into the world of experimentation. He emphasizes how fostering a culture of experimentation can lead to significant product innovations. The discussion covers common pitfalls in A/B testing, including sample ratio mismatches, and highlights the importance of trust and data integrity. Ronny also shares surprising results from his experiments and best practices for optimizing testing processes, making data-driven decisions essential for growth.

01:23:07

Creator website

Episode guests

Ronny Kohavi

AI Summary

Highlights

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Experimentation is crucial in driving product decisions and should be embraced even for high-risk, high-reward ideas.

Building an experiment-driven culture and prioritizing trust are essential for successful experimentation.

Small changes can have significant impacts, and incremental improvements should be pursued despite a high failure rate in experiments.

Deep dives

The Importance of Testing Everything

The speaker highlights the significance of conducting experiments to test any code change or feature introduction. Even small bug fixes or changes can have unexpected impacts. Experimentation is emphasized as a crucial practice, and high-risk, high-reward ideas should be pursued. The speaker advises being prepared for failure and emphasizes the value of testing big ideas, despite the likelihood of a high failure rate.

When should startups start running A.B. tests? 200.000 users

01:33

Experiment fail doesn’t mean the idea is wrong

00:50

The Importance of Investigating Results and Tuyman's Law

01:42

Amazon emails - Modeling the Cost of Spamming and unsubscribes

01:52

Intro

5min

The Power of A/B Testing: Small Changes, Big Impact

23min

Balancing Innovation and Experimentation

13min

The Strategic Approach to A/B Testing and Iteration

2min

Navigating Data-Driven Decision Making at Airbnb

12min

Unraveling A/B Testing Pitfalls

20min

Reflections on Chernobyl and Technical Interview Challenges

2min

Innovative Home Monitoring and Effective Team Communication

3min

The Hierarchy of Evidence and Real-Life A/B Testing

3min

Brought to you by Mixpanel—Event analytics that everyone can trust, use, and afford | Round—The private network built by tech leaders for tech leaders | Eppo—Run reliable, impactful experiments

—

Ronny Kohavi, PhD, is a consultant, teacher, and leading expert on the art and science of A/B testing. Previously, Ronny was Vice President and Technical Fellow at Airbnb, Technical Fellow and corporate VP at Microsoft (where he led the Experimentation Platform team), and Director of Data Mining and Personalization at Amazon. He was also honored with a lifetime achievement award by the Experimentation Culture Awards in September 2020 and teaches a popular course on experimentation on Maven. In today’s podcast, we discuss:

• How to foster a culture of experimentation

• How to avoid common pitfalls and misconceptions when running experiments

• His most surprising experiment results

• The critical role of trust in running successful experiments

• When not to A/B test something

• Best practices for helping your tests run faster

• The future of experimentation

—

Enroll in Ronny’s Maven class: Accelerating Innovation with A/B Testing at https://bit.ly/ABClassLenny. Promo code “LENNYAB” will give $500 off the class for the first 10 people to use it.

—

Find the full transcript at: https://www.lennysnewsletter.com/p/the-ultimate-guide-to-ab-testing

—

Where to find Ronny Kohavi:

• Twitter: https://twitter.com/ronnyk

• LinkedIn: https://www.linkedin.com/in/ronnyk/

• Website: http://ai.stanford.edu/~ronnyk/

—

Where to find Lenny:

• Newsletter: https://www.lennysnewsletter.com

• Twitter: https://twitter.com/lennysan

• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/