Data Engineering Podcast Your Data, Your Lake: How Observe Uses Iceberg and Streaming ETL for Observability
43 snips
Jan 18, 2026 Jacob Leverich, Cofounder and CTO of Observe, brings his vast experience from Splunk and Google to discuss the transformative power of lakehouse architectures in observability. He addresses the struggles organizations face with fragmented tools and high costs, introducing innovative solutions leveraging OpenTelemetry and Kafka for efficient data ingestion. Jacob dives into the benefits of using Iceberg for better data organization, the intricacies of query orchestration for low-latency responses, and the importance of metadata in enhancing user experience.
AI Snips
Chapters
Transcript
Episode notes
Observability Is Multi-Modal And Fragmented
- Observability combines logs, metrics, and traces to answer what is happening across distributed systems.
- Fragmented best-of-breed tools create silos and prevent a single team from answering incidents quickly.
Buffer Telemetry Into Kafka First
- Buffer telemetry into Kafka before loading the lake to provide durability and efficient batch writes.
- Tune batch size and timers to balance ingestion latency and throughput for interactive troubleshooting.
Make Lakehouses Interactive With Curation
- Lakehouses can be fast enough for observability if you curate and columnarize data by use case.
- Query execution can be optimized by splitting queries (cursoring) to return early interactive results.