PRODUCTEA with Leah, Growth & Senior Leadership

107: Andres Glusman: Why Your A/B Tests Are Lying (and How to Fix Them)

8 snips
May 20, 2025
Andres Glusman, an experimentation expert and former Meetup product leader, reveals some shocking truths about A/B testing. He explains why a staggering 89% of experiments fail, emphasizing that failure can lead to growth. For companies with low traffic, he advocates for bold changes over timid tweaks to ensure impactful results. He also candidly discusses the common pitfalls of redesigns and how they can backfire. Plus, he shares fresh perspectives on managing experiment data and recommends must-read books for product teams looking to innovate.
Ask episode
AI Snips
Chapters
Books
Transcript
Episode notes
INSIGHT

High Failure Rate Means Success

  • 89% of A/B experiments fail to move the needle.
  • A 65% loss rate means you're outperforming most companies by 3x.
ADVICE

Bundle Changes for Low Traffic

  • For companies with low traffic, test bigger changes rather than timid tweaks.
  • Bundle related changes that align with a bigger hypothesis to get measurable impact.
ADVICE

Ship When Stats Don't Stabilize

  • If an experiment on a big site shows no statistical significance after months, consider shipping it anyway.
  • Sampling challenges and user familiarity can make pure A/B testing misleading in large, living products.
Get the Snipd Podcast app to discover more snips from this episode
Get the app