High Signal cover image

High Signal

Episode 4: How to Build an Experimentation Machine and Where Most Go Wrong

Nov 7, 2024
Ramesh Johari, a Professor at Stanford University, dives into the evolution of online experimentation, especially for tech companies and marketplaces. He discusses how organizations can shift to self-learning models and the common pitfalls they encounter, such as risk aversion. The conversation touches on the transformative impact of generative AI on experimentation processes. Ramesh also shares strategies for cultivating a culture of learning from failure and integrating data scientists to enhance business value, all while moving beyond traditional A/B testing methods.
51:16

Episode guests

Podcast summary created with Snipd AI

Quick takeaways

  • Organizations must evolve their experimentation practices from basic A-B testing to innovative, risk-embracing strategies that foster learning from all outcomes.
  • Prioritizing data quality is essential for effective experimentation, as better data enables organizations to derive more significant insights and make informed decisions.

Deep dives

The Evolution of Online Experimentation

Organizations are evolving their experimentation practices to become faster, more adaptive, and self-learning. Initial experimentation often involves basic tests like A-B testing, where businesses simply try to determine which version of a product is better. However, as these organizations mature, they are shifting towards more innovative practices that encourage riskier testing and emphasize learning from both successes and failures. This evolution helps organizations not only identify winners but also understand the rationale behind outcomes, leading to more strategic decision-making.

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.
App store bannerPlay store banner

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode