My guest today is Ion Stoica, professor of computer science at UC Berkeley and the co-founder of Conviva, Databricks, and Anyscale. Over the last two decades, Ion’s research labs - the AMP Lab, the RISE Lab, and now the Sky Computing Lab - have seeded a generation of category-defining companies.

Ion has the unique ability to turn non-consensus ideas into durable businesses. He applied machine learning to video optimization with Conviva before AI became mainstream. He scaled Apache Spark into a $60B platform with Databricks. And now, with Anyscale, he’s betting on Ray as the foundation for distributed AI workloads.

In this episode, we dig into both sides of Ion’s work: how to build world-class research labs, and how to turn research into real companies. His clarity of thought makes the future feel legible, and his track record suggests he’s very often right.

Hope you enjoy the conversation!

Chapters:

00:00 The Spark thesis: win the ecosystem first, monetize later
01:00 Intro: From lab to company - Ion’s repeatable playbook
03:00 Did you always plan to become a founder, or did it just happen?
05:23 Let’s start with Spark - how did the project come about?
13:04 What were the most important early decisions at Databricks?
23:49 You were the first CEO - what did you have to learn (or unlearn)?
30:01 How was building Anyscale different from building Databricks?
33:53 What’s obvious to you about the future of AI that others miss?
37:31 Why AI works so well for code
41:00 The thesis behind OPAQUE Systems
44:06 Future infra will be heterogeneous, distributed, and vertically integrated
49:03 China’s edge: faster diffusion from lab to market
53:19 Platform companies still work, but only with the right investors
55:57 What role did the Databricks Unit (DBU) play in value capture?
58:02 AI progress is plateauing, but adoption is just beginning

MATT HOGAN: DARKNESS INTO LIGHT: EP. 156