PRODUCTEA with Leah, Growth & Senior Leadership 107: Andres Glusman: Why Your A/B Tests Are Lying (and How to Fix Them)
8 snips
May 20, 2025 Andres Glusman, an experimentation expert and former Meetup product leader, reveals some shocking truths about A/B testing. He explains why a staggering 89% of experiments fail, emphasizing that failure can lead to growth. For companies with low traffic, he advocates for bold changes over timid tweaks to ensure impactful results. He also candidly discusses the common pitfalls of redesigns and how they can backfire. Plus, he shares fresh perspectives on managing experiment data and recommends must-read books for product teams looking to innovate.
AI Snips
Chapters
Books
Transcript
Episode notes
High Failure Rate Means Success
- 89% of A/B experiments fail to move the needle.
- A 65% loss rate means you're outperforming most companies by 3x.
Bundle Changes for Low Traffic
- For companies with low traffic, test bigger changes rather than timid tweaks.
- Bundle related changes that align with a bigger hypothesis to get measurable impact.
Ship When Stats Don't Stabilize
- If an experiment on a big site shows no statistical significance after months, consider shipping it anyway.
- Sampling challenges and user familiarity can make pure A/B testing misleading in large, living products.
