Tech on the Rocks
Stream processing, LSMs and leaky abstractions with Chris Riccomini
Aug 23, 2024
Chris Riccomini, an expert in stream processing and LSMs, dives into the evolution of streaming systems, highlighting the challenges developers face. He critiques SQL's limitations in this space and emphasizes the need for better API designs. The discussion also touches on the impact of Rust on usability and efficiency, particularly in embedded libraries. Chris shares insights about his exciting project involving log-structured merge trees on object storage, and the future of data systems with a focus on composable databases and the importance of metadata in AI.
53:06
Episode guests
AI Summary
AI Chapters
Episode notes
Podcast summary created with Snipd AI
Quick takeaways
- Stream processing has evolved significantly, yet challenges remain in enhancing developer experience and the efficiency of production systems.
- The debate over SQL in streaming systems highlights the tension between accessibility for analytics engineers and the complications of leaky abstractions.
Deep dives
The Evolution of Stream Processing
Stream processing has seen significant advancements since its inception, particularly with the introduction of systems like Kafka. The speaker discusses how Kafka aimed to unify various data management practices, including log aggregation and change data capture, to enhance developer productivity within streaming environments. The goal was to simplify the complex nature of stream processing, which differs greatly from the straightforwardness of batch processing. Despite its progress, challenges still remain regarding developer experience and the efficiency of production systems utilizing stream processing.
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
