Complete Course: AI Experimentation
Product Growth Podcast
Analyzing Experimentation: Frequentist and Bayesian Approaches
This chapter examines the differences between frequentist and Bayesian statistics in experimental analysis, highlighting the market prevalence of frequentist methods. It also introduces Cupid, a technique designed for B2B contexts that utilizes existing user data to expedite results.
"AI has been the biggest driver of change in experimentation I've seen in my career."
That's Frederic De Todaro, Chief Product Officer at Kameleoon (profitable SaaS with 2K+ customers).
Fred has been at Kameleoon for 12+ years. In that role, he's helped thousands of teams use AI to experiment faster and smarter.
In today’s episode he’s breaking down:
How AI changes experimentation
How to experiment with AI features
Last week, I covered how one aspect of this: vibe experimentation. Today’s video is the A to Z AI impact.
If you experiment at work, this episode is for you.
----
Check out the conversation on Apple, Spotify and YouTube.
* Mobbin: Discover real-world design inspiration
* Jira Product Discovery: Build the right thing, reliably
* AI Product Strategy Certificate for Leaders: Get $550 off
----
Timestamps:
00:00 How AI Changed Experimentation Overview
01:54 The 4 Steps of Experimentation Framework
14:12 ADS
16:00 How AI has Changed Experimentation
21:08 User Behaviour Models
26:56 Multi-Armed Bandit vs Contextual Bandit
30:05 ADS
31:55 AI Content Genration
35:13 How Vibe Coding Changes Experimentation
41:35 Live Demo From Idea to Running Experiment in 2 Minutes
43:36 Two-Minute Build Achievement
51:49 How to Measure AI Features Properly
54:17 Measuring RAG Systems 3 Key Metrics
01:07:18 Best Experimentation Company Booking.com
01:10:10 Biggest PM Mistakes in Experimentation
01:13:52 Ending
----
Key Takeaways
1. Build is the bottleneck. Most teams can't A/B test because developers are busy. AI removes this constraint anyone can now create experiments in minutes using plain English.
2. 80% of experiments fail. But with AI opportunity detection, you can drill into failed experiments to find hidden wins, like features that work great on mobile but fail on desktop.
3. Vibe coding meets experimentation. It's not enough to build prototypes quickly. You need to test them with real users at scale. Prompt-based experimentation bridges this gap.
4. Context is everything. AI performs best when it has access to your website's framework, design system, and past experiments. The more context, the better the ideas and implementations.
5. Humans still matter. PMs bring business context, data scientists ensure statistical rigor, and AI handles the grunt work. It's augmentation, not replacement.
6. Start simple with feature flags. You don't need to copy Booking.com overnight. Begin with feature flags, then rollouts, then full experimentation. AI makes each step easier.
7. Measure beyond usage. For AI features, track: How many prompts to success? Time from idea to live? How often do developers step in? These reveal true value.
8. Multi-armed bandits for speed, contextual for personalization. Use multi-armed when you need quick answers. Use contextual when personalizing for each user.
9. Discovery and experimentation are partners. Discovery tells you what users say they want. Experimentation tells you what they actually do. You need both for the full picture.
----
Check out the conversation on Apple, Spotify and YouTube.
----
Related Podcasts:
* How to Build Things Faster as a Product Team
* Lessons from Super-Senior IC Experimentation PM
* Amplitude CEO: Demo, Story, and How They Build Product
-----
P.S. More than 85% of you aren't subscribed yet. If you can subscribe on YouTube, follow on Apple & Spotify, my commitment to you is that we'll continue making this content better.
----
If you want to advertise, email productgrowthppp at gmail.
This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.news.aakashg.com/subscribe
The AI-powered Podcast Player
