Evals: Establishing What 'Good' Looks Like

Jason Droege is the CEO of Scale AI, a company that provides foundational training data to every major AI lab. He previously co-founded Scour with Travis Kalanick and built Uber Eats from idea to $20 billion in revenue. In this conversation, Jason shares lessons from getting sued for $250 billion, discovering restaurant economics by weighing sandwich ingredients, and over 25 years of launching transformative technology businesses.

What you’ll learn:

What actually happened with Meta’s $14 billion investment in Scale AI

Why AI models still need human experts to improve, and how that relationship is evolving

How AI models learn from experts building websites and debugging code

The business lessons from building Uber Eats from zero to $20 billion

Why most enterprise data is useless for AI models today

Why urgent daily problems beat super-valuable occasional problems when building products

How to think independently when building new products and businesses

—

Brought to you by:

Merge—The fastest way to ship 220+ integrations: http://merge.dev/lenny

Figma Make—A prompt-to-code tool for making ideas real: https://www.figma.com/lenny/

Mercury—The art of simplified finances: https://mercury.com/

—

Transcript: https://www.lennysnewsletter.com/p/first-interview-with-scale-ais-ceo-jason-droege

—

My biggest takeaways (for paid newsletter subscribers): https://www.lennysnewsletter.com/i/174979621/my-biggest-takeaways-from-this-conversation

—

Where to find Jason Droege:

• X: https://x.com/jdroege

• LinkedIn: https://www.linkedin.com/in/jasondroege/

—

Where to find Lenny:

• Newsletter: https://www.lennysnewsletter.com

• X: https://twitter.com/lennysan

• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/

—

In this episode, we cover:

(00:00) Introduction to Jason Droege

(06:01) Jason’s early career and lessons learned

(10:27) The current state of Scale AI

(12:37) The shift to expert data labeling

(17:02) Challenges and strategies in finding experts

(18:48) Reinforcement learning and AI environments

(28:18) The future of AI and human involvement

(31:21) The role of evals

(35:25) What AI models will look like in the next few years

(41:43) Building Uber Eats and understanding customer needs

(48:19) The importance of independent thinking

(50:45) Setting high standards for new businesses

(53:03) Exploring and selecting business ideas

(57:07) The McDonald’s story

(01:00:13) The role of gross margins in business feasibility

(01:04:49) Why Jason says, “Not losing is a precursor to winning”

(01:09:12) Hiring and building teams

(01:12:11) AI corner

(01:14:47) Lightning round and final thoughts

—

Referenced:

• Travis Kalanick on X: https://x.com/travisk

• Scour: https://en.wikipedia.org/wiki/Scour_Inc.

• Scale: https://scale.com/

• Alexandr Wang on X: https://x.com/alexandr_wang

• Why experts writing AI evals is creating the fastest-growing companies in history | Brendan Foody (CEO of Mercor): https://www.lennysnewsletter.com/p/experts-writing-ai-evals-brendan-foody

• Brendan Foody’s post on X about knowledge work changing: https://x.com/BrendanFoody/status/1970163503702188048

• MIT Finds 95% of GenAI Pilots Fail Because Companies Avoid Friction: https://www.forbes.com/sites/jasonsnyder/2025/08/26/mit-finds-95-of-genai-pilots-fail-because-companies-avoid-friction/

• Uber Eats: https://www.ubereats.com/

• Stephen Chau on X: https://x.com/thestephenchau

• a16z Podcast: https://a16z.com/podcasts/a16z-podcast/

• F1: The Movie: https://www.imdb.com/title/tt16311594/

• V03: https://v03ai.com/

• Careers at Scale: https://scale.com/careers

—

Recommended books:

• The Selfish Gene: https://www.amazon.com/Selfish-Gene-Anniversary-Introduction/dp/0199291152

• The Road Less Traveled: A New Psychology of Love, Traditional Values, and Spiritual Growth: https://www.amazon.com/Road-Less-Traveled-Timeless-Traditional/dp/0743243153/

• Good to Great: Why Some Companies Make the Leap . . . And Others Don’t: https://www.amazon.com/Good-Great-Some-Companies-Others/dp/0066620996

• Thinking, Fast and Slow: https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555/

—

Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com.

—

Lenny may be an investor in the companies discussed.