AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Tech on the Rocks
Stream processing, LSMs and leaky abstractions with Chris Riccomini
Aug 23, 2024
Chris Riccomini, an expert in stream processing and LSMs, dives into the evolution of streaming systems, highlighting the challenges developers face. He critiques SQL's limitations in this space and emphasizes the need for better API designs. The discussion also touches on the impact of Rust on usability and efficiency, particularly in embedded libraries. Chris shares insights about his exciting project involving log-structured merge trees on object storage, and the future of data systems with a focus on composable databases and the importance of metadata in AI.
53:06
Episode guests
AI Summary
AI Chapters
Episode notes
Podcast summary created with Snipd AI
Quick takeaways
- Stream processing has evolved significantly, yet challenges remain in enhancing developer experience and the efficiency of production systems.
- The debate over SQL in streaming systems highlights the tension between accessibility for analytics engineers and the complications of leaky abstractions.
Deep dives
The Evolution of Stream Processing
Stream processing has seen significant advancements since its inception, particularly with the introduction of systems like Kafka. The speaker discusses how Kafka aimed to unify various data management practices, including log aggregation and change data capture, to enhance developer productivity within streaming environments. The goal was to simplify the complex nature of stream processing, which differs greatly from the straightforwardness of batch processing. Despite its progress, challenges still remain regarding developer experience and the efficiency of production systems utilizing stream processing.
Get the Snipd
podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any
moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share
& Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode