Use Tokens To Experiment Communally

Dianne Penn is Head of Product for Anthropic’s AI Research and Labs teams. She joined in 2023 as Anthropic’s first technical product manager, when the entire product team was five engineers, and has since helped ship every model from Claude 2 through Fable, and helped incubate Claude Code, MCP, Skills, computer use, tool use, and reasoning. Before Anthropic, she helped build Alexa’s AI at Amazon and, before that, traded high-yield bonds at JP Morgan Chase.

In our in-depth conversation, we discuss:

1. What Anthropic’s early days were like

2. The inflection points that turned Anthropic from an underdog into the fastest-growing company in history

3. How exactly Claude got so good at coding

4. The eval-driven development loop her team is pioneering

5. How to find joy in AI when everything is moving this fast

6. Why Claude’s willingness to push back is key to its success

7. Where human judgment remains irreplaceable

—

Brought to you by:

WorkOS—Make your app enterprise-ready, with SSO, SCIM, RBAC, and more

Mercury—Radically different banking, now with Command

—

Episode transcript: https://www.lennysnewsletter.com/p/anthropics-first-technical-pm-on

—

Archive of all Lenny's Podcast transcripts: https://www.dropbox.com/scl/fo/yxi4s2w998p1gvtpu4193/AMdNPR8AOw0lMklwtnC0TrQ?rlkey=j06x0nipoti519e0xgm23zsn9&st=ahz0fj11&dl=0

—

Where to find Dianne Penn:

• LinkedIn: linkedin.com/in/dianne-na-penn

—

Where to find Lenny:

• Newsletter: https://www.lennysnewsletter.com

• X: https://twitter.com/lennysan

• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/

—

In this episode, we cover:

(00:00) Introduction

(02:31) Early Anthropic days

(08:55) Big milestones

(13:50) Inside the exponential

(20:02) Token maxing

(23:30) Anthropic Labs and the incubation model

(27:30) How the research role works

(31:35) How to become a top researcher

(35:18) Frontier model safeguards

(39:38) Hiring in the AI era

(44:16) Building an eval set

(47:48) Evals vs PRDs

(49:55) The importance of hands-on leadership

(52:46) Finding joy in AI

(58:10) How Dianne uses Claude

(01:01:05) Avoiding overreliance on AI

(01:03:50) The constitution that makes Claude better

(01:07:11) AI writing and verification

(01:11:40) Where human brains will continue to be valuable

(01:14:10) Navigating AI with kids

(01:16:26) Alignment, the future of the PM role, and burnout

(01:21:54) Lightning round and final thoughts

—

Referenced:

• Anthropic: https://www.anthropic.com

• Golden Gate Claude: https://www.anthropic.com/news/golden-gate-claude

• Dario Amodei’s website: https://darioamodei.com

• Scaling Laws and Interpretability of Learning from Repeated Data: https://www.anthropic.com/research/scaling-laws-and-interpretability-of-learning-from-repeated-data

• Tokenmaxxing: How Top Builders Use AI To Do The Work Of 400 Engineers: https://www.ycombinator.com/library/Pa-tokenmaxxing-how-top-builders-use-ai-to-do-the-work-of-400-engineers

• Garry Tan on X: https://x.com/garrytan

• Anthropic co-founder on quitting OpenAI, AGI predictions, $100M talent wars, 20% unemployment, and the nightmare scenarios keeping him up at night | Ben Mann: https://www.lennysnewsletter.com/p/anthropic-co-founder-benjamin-mann

• Anthropic’s CPO on what comes next | Mike Krieger (co-founder of Instagram): https://www.lennysnewsletter.com/p/anthropics-cpo-heres-what-comes-next

• Introducing Labs: https://www.anthropic.com/news/introducing-anthropic-labs

• Louis CK | about airplane Wi Fi: https://www.youtube.com/watch?v=me4BZBsHwZs

• What happens after coding is solved? | Fiona Fung (Manager of the Claude Code and Cowork Teams): https://www.lennysnewsletter.com/p/building-the-most-ai-pilled-engineering

• The Anthropic Hive Mind: https://steve-yegge.medium.com/the-anthropic-hive-mind-d01f768f3d7b

• How to build a company that withstands any era | Eric Ries, Lean Startup author: https://www.lennysnewsletter.com/p/how-to-build-a-company-that-withstands

• Fallout on Prime Video: https://www.amazon.com/dp/B0CN4GGGQ2

• Fallout (video game): https://fallout.bethesda.net

• Claude Tag: https://www.anthropic.com/news/introducing-claude-tag

—

Recommended books:

• Crucial Conversations: Tools for Talking When Stakes Are High: https://www.amazon.com/dp/0071771328

• How to Raise an Adult: Break Free of the Overparenting Trap and Prepare Your Kid for Success: https://www.amazon.com/How-Raise-Adult-Overparenting-Prepare/dp/1627791779

• Incorruptible: Why Good Companies Go Bad... and How Great Companies Stay Great: https://www.amazon.com/dp/B0FWZZBPZB

—

Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].

—

Lenny may be an investor in the companies discussed.