The Bazaar in the Cathedral with Matei Zaharia, CTO & Co-founder at Databricks and Creator of Apache Spark

May 24, 2023

49:47

forum

Ask episode

view_agenda

Chapters

auto_awesome

Transcript

info_circle

Episode notes

When building a data platform, it’s important to stay true to your vision. Whether that's through creating a definitive user experience or an open platform that allows people to build upon it, you’re constructing a cathedral. This cathedral is sophisticated and dependable, and allows for a bazaar of business intelligence, machine learning, and AI use cases.

In this episode, Satyen interviews Matei Zaharia, Chief Technologist and Co-founder of Databricks. Matei is an open source trailblazer and the creator of Apache Spark, a widely used framework for distributed data processing. He is also an Associate Professor of Computer Science at Stanford University where he leads various data management and machine learning projects. Matei and Satyen discuss the Databricks and Alation partnership, exploring how platforms can help companies own their data, and consider the value of democratizing open source large language models.

--------

“One of the early stories about open source has been this thing about the cathedral and the bazaar. The cathedral is the thing that's all designed by one person, maybe. It's extremely coherent and so on, but also takes forever to build. And when you go there, there's one message you're hearing. And then the bazaar is the open thing. You don't know who's going to show up each day, but there'll be some really interesting goods and things that you just wouldn't see anywhere else. If you just want to get started and get stuff done, follow the defaults in the product and it'll work. But, we want to be open to some of that innovation and let people bring that in.” – Matei Zaharia

--------

Time Stamps:

*(01:33): The story behind Spark

*(11:56): Solving for user problems versus product vision

*(20:12): The cathedral and the bazaar of open source

*(24:04): Matei explains the Databricks Unity Catalog

*(31:04): The Databricks and Alation partnership

*(43:36): The data culture at Databricks

*(48:21): Satyen’s Takeaways

--------

Sponsor

This podcast is presented by Alation.