Experiment Nation: The Podcast

In this episode of Experiment Nation, host Jason Gossit sits down with Ameliano Blanca, a senior growth and product leader with 10+ years of experience driving revenue through data-driven experimentation.They dive deep into:How to build experimentation programs from the ground upThe balance between qualitative and quantitative insightsExperimentation in startups vs. mature organizationsCRO frameworks and the power of iterationHow to avoid "random testing mode" and adopt structured frameworksIf you’re a growth, product, or CRO professional, this episode is packed with actionable insights to help you scale experimentation programs, improve conversion rates, and create sustainable growth strategies.👉 Get our new book: Prove It or Lose It – The (Mostly) No-Nonsense Guide to Surviving Experimentation Program Drama at experimentnation.com and all major bookstores.📌 Chapters:0:00 – Why tests should be rerun after 12 months0:19 – Welcome & introduction with Jason Gossit0:35 – Meet Ameliano Blanca: growth & product leader1:32 – First steps in building experimentation programs2:42 – Measuring conversion funnels & revenue impact3:46 – Tools for gathering quantitative & qualitative insights5:05 – Heatmaps, surveys, and finding hidden gems6:00 – Drafting test ideas & prioritization7:19 – Using and adjusting the ICE framework9:50 – The “three strikes” rule for failed experiments11:16 – Experimentation in startups vs. mature companies13:00 – Why rerunning startup tests is critical14:57 – When to pause or rerun losing experiments15:29 – Complementing quantitative data with customer interviews17:16 – Running experiments at scale in larger organizations18:18 – Balancing speed vs. rigor in CRO testing20:04 – Defaulting experiments & tracking long-term results20:11 – CRO frameworks and the power of iteration22:26 – Why iteration is underrated in experimentation23:44 – Adapting frameworks to your company & industry25:18 – Advice for teams stuck in random testing mode27:39 – How CRO perspectives have evolved over 10 years28:39 – Growth, insights, and the rise of experimentation in strategy30:38 – Real-world example: financing simulator boosts conversions

In this episode of Experiment Nation, host Jason sits down with Namata Agarwal, a seasoned product designer with over 17 years of experience building digital products, websites, and apps.Together, they explore:- Ethical testing: how to balance user needs with business goals.- Leaks vs. puddles framework: a practical way to identify urgent vs. non-urgent product problems.- The role of design in CRO and experimentation.- Frameworks, guardrails, and collaboration strategies for building an experimentation culture.Whether you’re a CRO specialist, product manager, or experimentation nerd, this episode is packed with actionable insights you can bring to your own team.👉 Don’t forget to like, comment, and subscribe to stay updated on the latest in experimentation and growth!📖 Check out Prove It or Lose It: The (Mostly) No-Nonsense Guide to Surviving Experimentation Program Drama .⏱️ Chapters with Timestamps0:00 – Why design must be involved early0:22 – Welcome & intro to Jason0:43 – Guest intro: Namata Agarwal0:55 – Topics: Ethical testing & Leaks vs. Puddles1:18 – What is ethical testing?2:22 – Balancing business needs vs. user needs2:54 – Handling stakeholder pushback4:01 – Misaligned metrics and real solutions5:17 – The Leaks vs. Puddles analogy explained6:32 – Why better hypothesis building matters7:48 – Using Airtable & frameworks for test prioritization8:40 – Why experimentation must be collaborative9:24 – Reframing experiments around the user journey10:00 – Unpredictable user journeys & design challenges11:12 – Involving guest collaborators in experimentation11:50 – Mapping user journeys with Miro & qualitative research12:50 – Jobs To Be Done & other qualitative frameworks14:44 – Turning qualitative insights into hypotheses15:32 – How product design elevates CRO strategies16:11 – The importance of forms in experimentation17:36 – Why design should be included from day one18:20 – Design is not just visuals—it’s about impact19:11 – Governance, responsibility, and inclusivity in testing19:46 – Leak vs. Puddle framework deep dive22:58 – Running collaborative workshops25:45 – The “Stinky Fish” workshop format27:21 – Micro vs. macro frameworks for problem-solving28:05 – Balancing bold ideas vs. safe bets29:19 – Planning tests like an investment portfolio30:13 – Testing across 13 sites at scale30:26 – What Namata would A/B test in daily life31:57 – Art, painting, and creative balance32:42 – Martial arts & keeping calm32:56 – Wrap up & final thoughts33:14 – How to connect with Namata

In this episode of the Experiment Nation Podcast, host Ward Vanestra sits down with Ryan Lux, one of the first team members at Eppo and now Experimentation Evangelist at Datadog.They dive into:What Eppo is and how it helps companies experiment at scaleWhy data warehouses are critical for trustworthy resultsThe recent Datadog acquisition of Eppo and what it means for the futureAdvanced methods like CUPED (noise-cancelling for experiments) and contextual banditsHow to scale experimentation culture and get promoted by driving adoption, efficiency, and trustworthinessWhether you’re a CRO, growth leader, or experimentation enthusiast, this episode is packed with insights from someone who’s been shaping experimentation for over a decade.📘 Check out Experiment Nation’s new book: Prove It or Lose It: The Mostly No-Nonsense Guide to Surviving Experimentation Program Drama → experimentnation.com⏱️ Chapters0:00 – Why win rates don’t really increase0:20 – Introduction to Ryan Lucht & Eppo1:20 – What Eppo does and who it’s for2:18 – Experimentation at smaller companies3:18 – Datadog’s acquisition of Eppo3:59 – The data warehouse advantage6:03 – CUPED explained: noise-cancelling for experiments9:13 – Segments, personalization & pitfalls11:09 – Heterogeneous treatment effects (HTEs)12:06 – Personalization with contextual bandits14:00 – Why Datadog acquired Eppo16:22 – Data observability & product analytics17:52 – Experimentation and site performance19:46 – Content, culture, and experimentation career paths22:10 – Every experiment is valuable (win, lose, or flat)24:17 – OKRs for experimentation leaders25:47 – Three key OKRs: adoption, efficiency, trustworthiness28:14 – Knowledge building and learning plans29:34 – The vision: experiments as the ultimate insight system30:24 – Ryan’s Substack “Everything is an Experiment”31:02 – Closing thoughts & where to connect

In this conversation, Manuel Da Costa, founder of Effective Experiments and Efestra, shares his 15 years of expertise in the experimentation field. He highlights the crucial 'Trust Gap' between practitioners and decision-makers, emphasizing that running more experiments isn't the goal—making impactful decisions is. Manuel introduces the Learning Loop concept and offers strategies to bridge this divide. He argues for the need of experimentation leaders to be heard as strategic advisors, advocating for meaningful insights that drive real business transformations.

Paul Randall, Associate Director of Strategy at Speero, brings nearly 20 years of experience in conversion rate optimization and experimentation. He discusses the vital shift from CRO to an experiment-led growth strategy. Topics include building effective experimentation systems, the importance of cultural buy-in, and making informed growth bets. Randall also highlights the significance of blending qualitative and quantitative insights and the rise of EXL meetups, fostering a vibrant community for collaboration and sharing best practices.

Achille Vaulot, a seasoned expert in Conversion Rate Optimization (CRO), shares insights on building a mature CRO program. He discusses the evolution from basic A/B testing to a comprehensive, data-driven approach that includes thorough analysis and effective client communication. Achille highlights the importance of aligning CRO strategies with broader business objectives and fosters an experimentation culture. He also emphasizes the significance of qualitative data and the role of education in managing experiments, ensuring both successful and unsuccessful outcomes are leveraged for growth.

Ekaterina (Shpadareva) Gamsriegler, a seasoned growth and marketing expert, shared her insights on mobile app experimentation. Here are 5 key takeaways: Mobile experimentation is more complex: Release cycles are longer, and dependencies on app versions can slow down learning. Think beyond the app: Experimentation should encompass ads, store listing pages, and the app itself. Borrow insights from other areas: Use data from paid advertising to inform store listing optimizations. Interconnectivity is key: Changes in one area can impact others. Strong stakeholder management is crucial. Document everything: Detailed documentation helps track results and avoid surprises. Check out the full conversation on Experiment Nation’s YouTube channel!LinkedIn: https://www.linkedin.com/in/ekaterina-shpadareva-gamsriegler/ #mobilemarketing #appmarketing #experimentation #growthmarketing #experimentnation

Tracy Laranjo shares learnings from years of growing her career with Suchita Srinivasan, here are some key takeaways: 1. CRO vs Experimentation: CRO focuses on driving performance and results, while experimentation centers around decision-making and validating ideas. 2. Agency vs In-House: Tracy’s journey highlights the benefits of agency experience for rapid skill development and exposure to diverse challenges. 3. Building Your Experimental Brand: Don’t be afraid to be yourself online, even if it’s not the typical thought leader persona. 4. Adapting to Constant Change: Focus on fundamentals and building your skills; stay curious and open to new developments, but don’t fear the future. 5. CRO is for Everyone: While AB testing requires scale, CRO encompasses qualitative research and user interviews, making it accessible for businesses of all sizes.

Dr. Else van der Berg, a seasoned product management consultant for B2B SaaS startups, dives deep into the art of defining a narrow Ideal Customer Profile (ICP). She shares her layered approach to testing methods, emphasizing the blend of interviews and user tests. The discussion highlights the necessity of empathy in interpreting user feedback and the importance of evidence-based insights. Dr. van der Berg also stresses seeking mentorship in product management, urging aspiring professionals to connect for valuable guidance on their career journey.

Rishi Rawat, a Shopify product expert specializing in conversion rate optimization, shares his insights on maximizing e-commerce success. He emphasizes that product pages are crucial for conversions, acting as decision-making hubs. Rishi dispels the myth of pop-ups being effective and advocates for detailed product descriptions, arguing that content length can enhance clarity. He stresses building trust early on and understanding customer skepticism. Tune in for practical tips to boost your online sales and create engaging shopping experiences!