Elad Eldor, author of 'Kafka Troubleshooting in Production' and a Data Ops Engineer at Unity, shares his wealth of knowledge about Kafka. He discusses the differences between running Kafka on-prem and in the cloud, unveiling the complexities of cluster management and performance tuning. Elad emphasizes the importance of understanding system bottlenecks and resource optimization to avoid excessive costs. He also touches on the challenges of manual monitoring and the interplay between human expertise and technology in today’s operational landscape.

Elad Eldor, author of 'Kafka: Troubleshooting in Production', discusses the challenges of operating Kafka at scale and ways to mitigate potential issues. Topics include the importance of Kafka in the data pipeline, doubling retention in Kafka, managed vs. self-managed Kafka clusters, data lake complexity, monitoring for Kafka, troubleshooting unreplicated partitions, the cost of running Kafka in the cloud, and the need for a correlation tool.