Evals Define Better and Worse Outputs

Christian Idiodi sits down with Teresa Torres, author of "Continuous Discovery Habits" and host of the "Just Now Possible" podcast, for a candid conversation about what 15 months of building AI products has changed in her thinking, and what it hasn't. Teresa explains why she started out refusing to let AI anywhere near discovery, and what flipped her: a broken ankle, a research finding that AI raises the floor for novices and the ceiling for experts, and the realization that Claude learned interview synthesis faster than any human she'd taught.

–

Today's discussion unpacks:
• Why AI raises the floor for novices and the ceiling for experts
• The case for an AI interviewer, and its hard limit
• Synthetic users vs. digital twins: two very different bets
• What it costs an individual to be the change maker

–

References:
• AI Evals: A Hands-on Guide for Product Teams: https://www.producttalk.org/ai-evals/
• Customer Interview Analysis: Where AI Helps and Hurts: https://www.producttalk.org/customer-interview-analysis-ai/
• Ellen Brandenberger: https://www.linkedin.com/in/ellenbrandenberger/
• Hamel Husain: https://www.linkedin.com/in/hamelhusain/
• Indi Young: https://indiyoung.com/
• Marty Cagan: https://www.svpg.com/team/marty-cagan/
• Maven, AI Evals for Engineers & PMs: https://maven.com/parlance-labs/evals
• NotebookLM: https://notebooklm.google.com/
• Rob Fitzpatrick, The Mom Test: https://www.momtestbook.com/
• Shreya Shankar: https://www.linkedin.com/in/shrshnk/
• Steve Portigal: https://portigal.com/
• The Call to Adventure: https://www.producttalk.org/the-call-to-adventure/
• William Bridges Associates: https://wmbridges.com/

–

SVPG on discovery:
• Product Discovery Articles: https://www.svpg.com/insights/product-discovery-articles/
• Product Discovery Series: https://www.svpg.com/product-discovery-series/
• Product Discovery: Pitfalls and Anti-Patterns: https://www.svpg.com/product-discovery-anti-patterns/
• The AI Productivity Paradox: https://www.svpg.com/the-ai-productivity-paradox/

–

Where to find Teresa Torres:
• LinkedIn: https://www.linkedin.com/in/teresatorres/
• Product Talk: https://www.producttalk.org/
• Just Now Possible podcast: https://www.producttalk.org/just-now-possible/
• Vistaly: https://vistaly.com/

–

Where to find Christian Idiodi:
• LinkedIn: https://www.linkedin.com/in/cidiodi/
• SVPG profile: https://www.svpg.com/team/christian-idiodi/

–

Where to find SVPG:
• SVPG's newest book, TRANSFORMED: https://www.svpg.com/books/transformed/
• Other books: https://www.svpg.com/books/
• Articles: https://www.svpg.com/articles/
• Workshops: https://www.svpg.com/
• Videos: https://www.svpg.com/videos/

–

Timestamps:
00:00 Introduction
01:10 What Teresa believed about AI 15 months ago
03:40 A broken ankle and opening Pandora's box
05:20 Raising the floor and raising the ceiling
07:00 Where AI accelerates discovery, and where it doesn't
09:40 Judgment, taste, and why aesthetics isn't enough
12:44 Product is decision-making, and AI hasn't changed that
14:35 Did we spend 20 years fixing the wrong bottleneck?
16:08 What hundreds of "customer interviews" actually look like
17:34 The case for an AI interviewer
18:44 Synthetic users, digital twins, and what still doesn't work
23:10 Why Teresa started Just Now Possible
26:35 Better and worse, not right and wrong
28:20 Evals: defining what good looks like as a team
30:32 The ethics of looking at AI traces
33:20 Why AI transformations don't transform organizations
36:05 What teams should never outsource to AI
41:25 Choosing to be a beginner again

–

Disclaimer:

While this podcast is named "Product Therapy", it is not hosted by licensed therapists or mental health professionals, and it is in no way a substitute for professional mental health services. We recognize the importance of mental well-being and encourage anyone facing personal difficulties to seek support from qualified professionals. See https://findahelpline.com/

–

Production:
Production by supermix.io