Evaluation Failure: Studio Sound Lessons

Today’s Episode

Laura Burkhauser started as an IC PM at Descript, the $550M AI video editing platform.

Three years later, she’s CEO.

She shipped AI features that worked.

In today’s episode, you’ll hear the exact features Laura built, the eval framework she used, and her complete IC → CEO path.

This might be the most actionable career episode you listen to all year.

----

Check out the conversation on Apple, Spotify and YouTube.

Brought to you by:

* Maven: Improve your PM skills with awesome courses

* Pendo: #1 Software Experience Management Platform

* Vanta: Automate compliance across 35+ frameworks like SOC 2 and ISO 27001

* NayaOne: Airgapped cloud-agnostic sandbox

* Kameleoon: Leading AI experimentation platform

----

Key Takeaways1. Map your user journey BEFORE picking AI features. DeScript identified pain points (retakes, eye contact, rambling), then asked "what just became possible with LLMs?" Build that intersection.2. Build prepackaged buttons, not blank chat boxes. Each DeScript AI tool is a carefully crafted prompt behind a single button that delivers reliable results every time.3. Use human evals on production data before shipping. Test on real customer data, ask "would I use this as a customer?" If yes, ship. If no, don't.4. The ultimate metric is export rate. If users apply your AI feature then remove it before exporting, it didn't meet their quality bar.5. Switch from buttons to chat when you hit 30+ parameters. When users wanted topic selection, speaker choice, and platform optimization, chat became better than buttons.6. Match your eval data to actual use case. DeScript failed with Studio Sound because they tested on terrible audio (vacuuming, jackhammers) when real users had laptop microphones. Different models handle different quality levels.7. Test agents with real customer language early. Don't use toy data or employee terminology. Mix sophistication levels—some advanced at video and AI, some complete beginners—to understand how real people prompt.8. Launch AI agents to new users first. Video editing is hard and many people quit. DeScript tested Underlord on activation and it won, so new users got it first before existing users.9. Choose breadth over depth for product-wide agents. DeScript chose breadth—Underlord works across all features because "we're not a point solution." Requires more context, tool coverage, and evals but serves the product vision.10. Earn founder trust by getting command, not by being strategic. Use the product extensively. Talk to customers constantly. When you speak, people think "Smart" and invite you to more rooms. Ship features before focusing on strategy.

----

Where to Find Laura Burkhauser

* LinkedIn

* Company

----

Related Content

Podcasts:

* AI Product Leadership Masterclass

* Conversation with the CEO and Founder of Bolt

* This $20M AI Founder Is Challenging Elon and Sam Altman | Roy Lee, Cluely

Newsletters:

* Ultimate Guide to AI Prototyping Tools

* AI Agents: The Ultimate Guide for PMs

* How to Land a $300K+ AI Product Manager Job

PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.news.aakashg.com/subscribe