Reinventing Kafka on object storage (Changelog Interviews #606)

Aug 29, 2024

Ryan Worl, Co-founder and CTO at WarpStream, shares insights on redefining Kafka for cloud environments using object storage. He discusses Kafka's evolution, its widespread use, and WarpStream's innovative approach, sparked by a compelling blog post. The chat dives into the complexities of managing Kafka on-premises versus in the cloud, and how WarpStream aims to simplify data streaming with modern solutions. Ryan also touches on the challenges of commercial open source, emphasizing user feedback and competitive strategies in tech development.

Ask episode

Chapters

Transcript

Episode notes

Intro

00:00 • 2min

Streamlining Enterprise-Grade SDK Development with OpenAPI

01:50 • 3min

Exploring Kafka's Role in Warpstream's Technology

04:36 • 3min

Kafka's Complex Landscape

07:33 • 15min

Innovations in Data Management: Husky and Beyond

22:58 • 10min

Navigating Storage Solutions in Application Design

33:20 • 22min

Navigating Kafka Compatibility and Market Dynamics

55:25 • 21min

Navigating Integration Challenges in B2B SaaS

01:16:50 • 10min

Navigating Commercial Open Source Challenges

01:26:52 • 15min

Reimagining Kafka for Cloud Environments

01:41:55 • 2min

Ryan Worl, Co-founder and CTO at WarpStream, joins us to talk about the world of Kafka and data streaming and how WarpStream redesigned the idea of Kafka to run in modern cloud environments directly on top of object storage. Last year they posted a blog titled, “Kafka is dead, long live Kafka” that hit the top of Hacker News to put WarpStream on the map. We get the backstory on Kafka and why it’s so widely used, who created it and for what purpose, and the behind the scenes on all things WarpStream.

Join the discussion

Changelog++ members save 14 minutes on this episode because they made the ads disappear. Join today!

Sponsors:

Speakeasy – Production-ready, enterprise-resilient, best-in-class SDKs crafted in minutes. Speakeasy takes care of the entire SDK workflow to save you significant time, delivering SDKs to your customers in minutes with just a few clicks! Create your first SDK for free!
Supabase – Supabase just finished their 12th launch week! Check it out. Or get a month of Supabase Pro (FREE) by going to supabase.com/changelogpod
Paragon – Ship native integrations to production in days with more than 130 pre-built connectors, or configure your own custom integrations. Built for product and engineering. Learn more at useparagon.com/changelog
Unblocked – Other developer tools can’t tell you how your codebase works and why. Unblocked can. We augment your code with context from Slack, Confluence, Jira, and more, so you get accurate answers without having to search for them. Sign up for free at getunblocked.com

Featuring:

Ryan Worl – Website, GitHub, LinkedIn, X
Adam Stacoviak – Website, GitHub, LinkedIn, Mastodon, X
Jerod Santo – GitHub, LinkedIn, Mastodon, X

Show Notes:

Kafka is dead, long live Kafka
Apache Kafka - open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
Current 2024
Amazon Managed Streaming for Apache Kafka
Bobby Brown - My Prerogative

Something missing or broken? PRs welcome!