Experiment Nation: The Podcast

High-velocity experimentation can dramatically increase how quickly organizations learn, optimize their products, and drive revenue. But many teams still run A/B tests sequentially, waiting for one experiment to finish before starting the next.In this video, Varun, co-founder of Geeks of Digital, explains how overlapping A/B tests can remove experimentation bottlenecks while still maintaining rigorous randomization and measurement.You’ll learn how layered experimentation works, why the "isolation fallacy" can hold teams back, when experiments should still be isolated, and how organizations can build an operational system for running experiments at scale.The video also covers the "blue-by-blue" isolation trap, compounding gains from multiple winning experiments, fake door testing, experimentation culture, and how companies like Meta, Booking.com, Airbnb, and Spotify approach experimentation.---Music: “Strong Female Pop Attitude” by JuicyLucyMusic, licensed via AudioJungle/Envato Market.https://audiojungle.net/item/strong-pop-attitude/19345362 This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Edd Saunders, a CRO and product experimentation leader with 10+ years' experience and current Product Experimentation Manager at Job Leads. He talks about breaking into CRO without a traditional background. He covers agency vs in-house learning, winning developer buy-in, smarter prioritization beyond ICE, focusing on customer problems, and treating experiments as customer intelligence rather than just conversion lifts.

Jon Crowder joins Tracy Laranjo to explain why ethical experimentation is not just better for customers—it is better for long-term growth. They unpack fake urgency, misleading pricing and subscription tactics, short-term A/B test wins that erode trust, and practical ways to design experiments that improve retention, lifetime value, and revenue.Chapters:0:00 Welcome & introducing Jon Crowder1:29 Why ethical experimentation matters2:17 The pressure to drive results at any cost7:17 When winning an A/B test loses money long term8:34 Reframing urgency into trust-building experiences13:38 Choosing clients and setting ethical boundaries16:37 The ethics of price testing19:21 Fake sales, subscriptions & agency pressure20:46 How to steer unethical tests into ethical ones23:09 Have urgency and scarcity gone too far?24:02 Why legacy businesses cling to outdated tactics27:36 Gen Z, trust, and changing expectations29:48 What an ethical future for digital experiences looks like34:30 Brands that earn loyalty through values38:27 Why ethical experiences convert better40:15 A simple test for ethical experimentation41:37 Where to find Jon#CRO #Experimentation #ABTesting #GrowthMarketing #EthicalMarketing #ProductGrowth #ConversionOptimization #DigitalExperience #CustomerExperience #Retention #MarketingStrategy #GrowthStrategy This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

In this episode, we dive into CRO “best practices” that completely failed in real-world experimentation. From homepage carousels unexpectedly winning tests to why most experimenters still don’t trust AI-generated CRO workflows, we unpack the messy reality behind optimization advice.We also debate synthetic AI users for research, discuss the most useful survey questions in CRO, roast dark-mode websites, and explain why emotional branding can sometimes hurt usability.Join our slack: https://join.slack.com/t/experiment-nation/shared_invite/zt-3zowejvoj-ujS_tbqfXcPkS9O0rnWhTQTopics include:Why homepage carousels occasionally winAI-powered A/B testing platformsSynthetic users & fake research dataBest and worst CRO survey questionsDark mode website UXWhy qualitative context still mattersJoin the Experiment Nation Slack community to participate in future discussions and question threads.Chapters00:00 Intro & CRO community questions01:46 CRO best practices that failed in real life02:28 Do homepage carousels actually work?05:02 Hero videos, agencies & emotional design06:25 When carousels might make sense08:24 Should AI run the entire CRO workflow?11:48 Why experimenters still don’t trust AI14:49 Best & worst survey questions in CRO17:30 Emotional branding vs usable websites18:26 Dark mode websites & readability debate21:29 Synthetic AI users for testing & research23:56 Why fake users can’t mimic real humans26:32 Using AI as a research starting point28:41 Closing thoughts & Slack community invite This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

What do you do when an experiment tanks conversion rate by 8% in less than a week?In this episode, Gerda and Rommil break down real CRO and experimentation scenarios submitted by the community, including failed tests, executive pressure, guardrail metrics, knowledge management, developer friction, and why experimentation teams always seem to end up back in spreadsheets.Check out: http://katsed.com/We also discuss:* Root cause analysis for failed experiments* How to calm leadership during testing disasters* Guardrail metrics and control limits* Why experimentation knowledge banks become messy* Airtable vs spreadsheets vs enterprise tools* Why developers and designers struggle with experimentation culture* How to onboard technical teams into testing workflowsA practical conversation for experimentation leads, growth teams, product managers, CRO specialists, and anyone building an experimentation culture.Chapters00:00 Burger Menu Debate: Left vs Right01:28 Mystery Meat Navigation & UX History03:03 Why Modern Phones Break UX05:08 Community CRO Scenarios Begin06:18 Experiment Tanks Conversion by 8%07:33 Root Cause Analysis & Leadership Pressure09:17 QA, Tracking Errors & Failed Tests10:03 Building a CRO Analysis Tool12:18 Probabilistic Root Cause Detection14:26 Managing Panic During Bad Experiments15:21 Guardrail Metrics Explained16:40 Using Control Limits in Experimentation18:18 Building an Experimentation Knowledge Bank20:05 Why You Don’t Need AI for Everything21:08 Why Process Changes Create Admin Pain23:08 Migrating Between Experimentation Tools24:31 Why Teams Always Go Back to Spreadsheets26:50 CRO Teams vs Developers & Designers29:09 Why Early Experiments Should Be Simple30:41 Why Developers Hate “Throwaway Code”32:16 Designers vs Experiment Constraints33:02 Outro & Next Community Questions This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

They unpack why most teams call what they do research but miss the mark. They compare manual coding to AI for messy qualitative answers and explain when sampling makes sense. They dig into survey bias, organizational politics that stall research, and how open-ended replies fuel sharper marketing and personalization.

A playful start about photography leads into a frank look at CRO today, from AI noise making outputs feel the same to real anxiety around layoffs. They debate team structures — centralized, hybrid, embedded — and why experimentation is often labeled a nice-to-have. The conversation covers behaviours that wreck culture and practical ways individual contributors can spark change and reframe testing as growth systems.

Conversations about where AI helps experimentation and where it quietly breaks workflows. Practical debates on measuring MDE for continuous revenue metrics like ARPU. Tips for carving out time for customer research when schedules are overloaded. Warnings about unpaid interview audits and how to handle volume hiring for CRO roles. Real talk on escaping the endless ROI loop and why some agencies lack designers.

They debunk the biggest myths around A/B testing and why it cannot rescue broken businesses. They explain why a single-person CRO team is a risky bet. They explore why copying other tests often fails and when copying might make sense. They discuss how experimentation tools get chosen and the politics and buy-in that really matter. They share tips for working with clients and improving cross-team collaboration.

Conversations jump from recovering after work setbacks to battling burnout and rediscovering motivation. They debate why long-form content wins for complex topics and unpack CRO differences across B2B, SaaS, and ecommerce. The hidden buyer problem and redesigns without A/B testing get scrutinized. Practical paths for beginners and tactics to get others to pay attention to data are explored.