Berkeley professor Ion Stoica, co-founder of Databricks and Anyscale, transformed the open source projects Spark and Ray into successful AI infrastructure companies. He talks about what mattered most for Databricks' success -- the focus on making Spark win and making Databricks the best place to run Spark. He highlights the importance of striking key partnerships -- the Microsoft partnership in particular that accelerated Databricks' growth and contributed to Spark's dominance among data scientists and AI engineers. He also shares his perspective on finding new problems to work on, which holds lessons for aspiring founders and builders: 1) building systems in new areas that, if widely adopted, put you in the best position to understand the new problem space, and 2) focusing on a problem that is more important tomorrow than today.
Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital
Mentioned in this episode:
-
Spark: The open source platform for data engineering that Databricks was originally based on.
-
Ray: Open source framework to manage, executes and optimizes compute needs across AI workloads, now productized through Anyscale
-
MosaicML: Generative AI startups founded by Naveen Rao that Databricks acquired in 2023.
-
Unity Catalog: Data and AI governance solution from Databricks.
-
CIB Berkeley: Multi-strategy hedge fund at UC Berkeley that commercializes research in the UC system.
-
Hadoop: A long-time leading platform for large scale distributed computing.
-
VLLM and Chatbot Arena: Two of Ion’s students’ projects that he wanted to highlight.
