Developer Voices Making Apache Kafka Diskless (with Filip Yonov & Josep Prat)
14 snips
Jun 5, 2025 Joining the conversation are Josep Prat and Filip Yonov from Aiven, both experts in Apache Kafka and data streaming. They delve into the innovative shift towards making Kafka diskless, utilizing cloud services like S3 for scalable storage. The discussion includes the economic implications of data replication, the challenges of integrating new features without breaking existing compatibility, and the importance of community collaboration in open-source projects. Listeners will also find insights into Kafka's configurations and the future of cloud integration.
AI Snips
Chapters
Transcript
Episode notes
Object Storage Trade-Offs
- Replacing Kafka's storage layer with object storage sacrifices some latency for cost savings and infinite scalability.
- Despite the latency cost, this approach is commercially appealing and has led to significant business valuations.
Replication Costs in Multi-AZ Clouds
- Kafka replication across availability zones incurs high inter-zone data transfer costs due to multiple redundant connections.
- This replication model greatly increases operational expenses in multi-AZ cloud deployments.
Economic and Ergonomic Drivers
- The primary motivation for diskless Kafka development was economic, aiming to reduce operational costs.
- Additional discovered benefits include improved elasticity and better ergonomic features beyond just cost savings.