S5E3: Eppo, Datadog & The Future of Experimentation | Ryan Lucht on Advanced Testing & Culture

In this episode of the Experiment Nation Podcast, host Ward Vanestra sits down with Ryan Lux, one of the first team members at Eppo and now Experimentation Evangelist at Datadog.They dive into:What Eppo is and how it helps companies experiment at scaleWhy data warehouses are critical for trustworthy resultsThe recent Datadog acquisition of Eppo and what it means for the futureAdvanced methods like CUPED (noise-cancelling for experiments) and contextual banditsHow to scale experimentation culture and get promoted by driving adoption, efficiency, and trustworthinessWhether you’re a CRO, growth leader, or experimentation enthusiast, this episode is packed with insights from someone who’s been shaping experimentation for over a decade.📘 Check out Experiment Nation’s new book: Prove It or Lose It: The Mostly No-Nonsense Guide to Surviving Experimentation Program Drama → experimentnation.com⏱️ Chapters0:00 – Why win rates don’t really increase0:20 – Introduction to Ryan Lucht & Eppo1:20 – What Eppo does and who it’s for2:18 – Experimentation at smaller companies3:18 – Datadog’s acquisition of Eppo3:59 – The data warehouse advantage6:03 – CUPED explained: noise-cancelling for experiments9:13 – Segments, personalization & pitfalls11:09 – Heterogeneous treatment effects (HTEs)12:06 – Personalization with contextual bandits14:00 – Why Datadog acquired Eppo16:22 – Data observability & product analytics17:52 – Experimentation and site performance19:46 – Content, culture, and experimentation career paths22:10 – Every experiment is valuable (win, lose, or flat)24:17 – OKRs for experimentation leaders25:47 – Three key OKRs: adoption, efficiency, trustworthiness28:14 – Knowledge building and learning plans29:34 – The vision: experiments as the ultimate insight system30:24 – Ryan’s Substack “Everything is an Experiment”31:02 – Closing thoughts & where to connect