Experiment Nation: The Podcast

In this episode of the Experiment Nation Podcast, host Ward Vanestra sits down with Ryan Lux, one of the first team members at Eppo and now Experimentation Evangelist at Datadog.They dive into:What Eppo is and how it helps companies experiment at scaleWhy data warehouses are critical for trustworthy resultsThe recent Datadog acquisition of Eppo and what it means for the futureAdvanced methods like CUPED (noise-cancelling for experiments) and contextual banditsHow to scale experimentation culture and get promoted by driving adoption, efficiency, and trustworthinessWhether you’re a CRO, growth leader, or experimentation enthusiast, this episode is packed with insights from someone who’s been shaping experimentation for over a decade.📘 Check out Experiment Nation’s new book: Prove It or Lose It: The Mostly No-Nonsense Guide to Surviving Experimentation Program Drama → experimentnation.com⏱️ Chapters0:00 – Why win rates don’t really increase0:20 – Introduction to Ryan Lucht & Eppo1:20 – What Eppo does and who it’s for2:18 – Experimentation at smaller companies3:18 – Datadog’s acquisition of Eppo3:59 – The data warehouse advantage6:03 – CUPED explained: noise-cancelling for experiments9:13 – Segments, personalization & pitfalls11:09 – Heterogeneous treatment effects (HTEs)12:06 – Personalization with contextual bandits14:00 – Why Datadog acquired Eppo16:22 – Data observability & product analytics17:52 – Experimentation and site performance19:46 – Content, culture, and experimentation career paths22:10 – Every experiment is valuable (win, lose, or flat)24:17 – OKRs for experimentation leaders25:47 – Three key OKRs: adoption, efficiency, trustworthiness28:14 – Knowledge building and learning plans29:34 – The vision: experiments as the ultimate insight system30:24 – Ryan’s Substack “Everything is an Experiment”31:02 – Closing thoughts & where to connect

In this conversation, Manuel Da Costa, founder of Effective Experiments and Efestra, shares his 15 years of expertise in the experimentation field. He highlights the crucial 'Trust Gap' between practitioners and decision-makers, emphasizing that running more experiments isn't the goal—making impactful decisions is. Manuel introduces the Learning Loop concept and offers strategies to bridge this divide. He argues for the need of experimentation leaders to be heard as strategic advisors, advocating for meaningful insights that drive real business transformations.

Paul Randall, Associate Director of Strategy at Speero, brings nearly 20 years of experience in conversion rate optimization and experimentation. He discusses the vital shift from CRO to an experiment-led growth strategy. Topics include building effective experimentation systems, the importance of cultural buy-in, and making informed growth bets. Randall also highlights the significance of blending qualitative and quantitative insights and the rise of EXL meetups, fostering a vibrant community for collaboration and sharing best practices.

Achille Vaulot, a seasoned expert in Conversion Rate Optimization (CRO), shares insights on building a mature CRO program. He discusses the evolution from basic A/B testing to a comprehensive, data-driven approach that includes thorough analysis and effective client communication. Achille highlights the importance of aligning CRO strategies with broader business objectives and fosters an experimentation culture. He also emphasizes the significance of qualitative data and the role of education in managing experiments, ensuring both successful and unsuccessful outcomes are leveraged for growth.

Ekaterina (Shpadareva) Gamsriegler, a seasoned growth and marketing expert, shared her insights on mobile app experimentation. Here are 5 key takeaways: Mobile experimentation is more complex: Release cycles are longer, and dependencies on app versions can slow down learning. Think beyond the app: Experimentation should encompass ads, store listing pages, and the app itself. Borrow insights from other areas: Use data from paid advertising to inform store listing optimizations. Interconnectivity is key: Changes in one area can impact others. Strong stakeholder management is crucial. Document everything: Detailed documentation helps track results and avoid surprises. Check out the full conversation on Experiment Nation’s YouTube channel!LinkedIn: https://www.linkedin.com/in/ekaterina-shpadareva-gamsriegler/ #mobilemarketing #appmarketing #experimentation #growthmarketing #experimentnation

Tracy Laranjo shares learnings from years of growing her career with Suchita Srinivasan, here are some key takeaways: 1. CRO vs Experimentation: CRO focuses on driving performance and results, while experimentation centers around decision-making and validating ideas. 2. Agency vs In-House: Tracy’s journey highlights the benefits of agency experience for rapid skill development and exposure to diverse challenges. 3. Building Your Experimental Brand: Don’t be afraid to be yourself online, even if it’s not the typical thought leader persona. 4. Adapting to Constant Change: Focus on fundamentals and building your skills; stay curious and open to new developments, but don’t fear the future. 5. CRO is for Everyone: While AB testing requires scale, CRO encompasses qualitative research and user interviews, making it accessible for businesses of all sizes.

Dr. Else van der Berg, a seasoned product management consultant for B2B SaaS startups, dives deep into the art of defining a narrow Ideal Customer Profile (ICP). She shares her layered approach to testing methods, emphasizing the blend of interviews and user tests. The discussion highlights the necessity of empathy in interpreting user feedback and the importance of evidence-based insights. Dr. van der Berg also stresses seeking mentorship in product management, urging aspiring professionals to connect for valuable guidance on their career journey.

Rishi Rawat, a Shopify product expert specializing in conversion rate optimization, shares his insights on maximizing e-commerce success. He emphasizes that product pages are crucial for conversions, acting as decision-making hubs. Rishi dispels the myth of pop-ups being effective and advocates for detailed product descriptions, arguing that content length can enhance clarity. He stresses building trust early on and understanding customer skepticism. Tune in for practical tips to boost your online sales and create engaging shopping experiences!

Black Friday is coming, and the pressure is on. But don’t worry, Ilan Hurwitz has some expert advice to help you navigate the chaos. Here are five key takeaways from his chat with Tracy Laranjo: 1) Start now! Don’t wait until the last minute to get buy-in and start testing. 2) Speak the language of profit. Showcase the ROI of CRO to your stakeholders, focusing on increasing conversions and profits. 3) Don’t be afraid to experiment. Test offers, product placements, and even email campaigns outside of standard website A/B testing. 4) Focus on the customer journey. Don’t just look at your website. Analyze customer service tickets, reviews, and LTV to identify areas for improvement. 5) Think beyond discounting. Explore creative offers like gift with purchase, cross-sells, and subscription incentives to boost sales. Get ready to conquer Black Friday! #BlackFriday #CRO #ecommerce #marketing #experimentation

Lukas Vermeer, an expert in fostering a culture of experimentation, shares insights on cultivating innovation within organizations. He emphasizes that experimentation is a journey requiring time and commitment, not an overnight fix. The importance of cross-functional teams in breaking down silos is highlighted, along with the necessity of leadership buy-in to secure resources and support. Lukas also discusses the vital role of organizational processes in facilitating collaboration and encourages embracing the challenges that come with change.