Benchmarks can mislead model progress

Edwin Chen is the founder and CEO of Surge AI, the company that teaches AI what’s good vs. what’s bad, powering frontier labs with elite data, environments, and evaluations. Surge surpassed $1 billion in revenue with under 100 employees last year, completely bootstrapped—the fastest company in history to reach this milestone. Before founding Surge, Edwin was a research scientist at Google, Facebook, and Twitter and studied mathematics, computer science, and linguistics at MIT.

We discuss:

1. How Surge reached over $1 billion in revenue with fewer than 100 people by obsessing over quality

2. The story behind how Claude Code got so good at coding and writing

3. The problems with AI benchmarks and why they’re pushing AI in the wrong direction

4. How RL environments are the next frontier in AI training

5. Why Edwin believes we’re still a decade away from AGI

6. Why taste and human judgment shape which AI models become industry leaders

7. His contrarian approach to company building that rejects Silicon Valley’s “pivot and blitzscale” playbook

8. How AI models will become increasingly differentiated based on the values of the companies building them

—

Brought to you by:

Vanta—Automate compliance. Simplify security.

WorkOS—Modern identity platform for B2B SaaS, free up to 1 million MAUs

Coda—The all-in-one collaborative workspace

—

Transcript: https://www.lennysnewsletter.com/p/surge-ai-edwin-chen

—

My biggest takeaways (for paid newsletter subscribers): https://www.lennysnewsletter.com/i/180055059/my-biggest-takeaways-from-this-conversation

—

Where to find Edwin Chen:

• X: https://x.com/echen

• LinkedIn: https://www.linkedin.com/in/edwinzchen

• Surge’s blog: https://surgehq.ai/blog

—

Where to find Lenny:

• Newsletter: https://www.lennysnewsletter.com

• X: https://twitter.com/lennysan

• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/

—

In this episode, we cover:

(00:00) Introduction to Edwin Chen

(04:48) AI’s role in business efficiency

(07:08) Building a contrarian company

(08:55) An explanation of what Surge AI does

(09:36) The importance of high-quality data

(13:31) How Claude Code has stayed ahead

(17:37) Edwin’s skepticism toward benchmarks

(21:54) AGI timelines and industry trends

(28:33) The Silicon Valley machine

(33:07) Reinforcement learning and future AI training

(39:37) Understanding model trajectories

(41:11) How models have advanced and will continue to advance

(42:55) Adapting to industry needs

(44:39) Surge’s research approach

(48:07) Predictions for the next few years in AI

(50:43) What’s underhyped and overhyped in AI

(52:55) The story of founding Surge AI

(01:02:18) Lightning round and final thoughts

—

Referenced:

• Surge: https://surgehq.ai

• Surge’s product page: https://surgehq.ai/products

• Claude Code: https://www.claude.com/product/claude-code

• Gemini 3: https://aistudio.google.com/models/gemini-3

• Sora: https://openai.com/sora

• Terrence Rohan on LinkedIn: https://www.linkedin.com/in/terrencerohan

• Richard Sutton—Father of RL thinks LLMs are a dead end: https://www.dwarkesh.com/p/richard-sutton

• The Bitter Lesson: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

• Reinforcement learning: https://en.wikipedia.org/wiki/Reinforcement_learning

• Grok: https://grok.com

• Warren Buffett on X: https://x.com/WarrenBuffett

• OpenAI’s CPO on how AI changes must-have skills, moats, coding, startup playbooks, more | Kevin Weil (CPO at OpenAI, ex-Instagram, Twitter): https://www.lennysnewsletter.com/p/kevin-weil-open-ai

• Anthropic’s CPO on what comes next | Mike Krieger (co-founder of Instagram): https://www.lennysnewsletter.com/p/anthropics-cpo-heres-what-comes-next

• Brian Armstrong on LinkedIn: https://www.linkedin.com/in/barmstrong

• Interstellar on Prime Video: https://www.amazon.com/Interstellar-Matthew-McConaughey/dp/B00TU9UFTS

• Arrival on Prime Video: https://www.amazon.com/Arrival-Amy-Adams/dp/B01M2C4NP8

• Travelers on Netflix: https://www.netflix.com/title/80105699

• Waymo: https://waymo.com

• Soda versus pop: https://flowingdata.com/2012/07/09/soda-versus-pop-on-twitter

—

Recommended books:

• Stories of Your Life and Others: https://www.amazon.com/Stories-Your-Life-Others-Chiang/dp/1101972122

• The Myth of Sisyphus: https://www.amazon.com/Myth-Sisyphus-Vintage-International/dp/0525564454

• Le Ton Beau de Marot: In Praise of the Music of Language: https://www.amazon.com/dp/0465086454

• Gödel, Escher, Bach: An Eternal Golden Braid: https://www.amazon.com/G%C3%B6del-Escher-Bach-Eternal-Golden/dp/0465026567

—

Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com.

—

Lenny may be an investor in the companies discussed.