The Data Engineering Show How Zendesk engineers manage customer-facing data applications
7 snips
Feb 17, 2022 Ananth Packkildurai, Principal Software Engineer at Zendesk and curator of the popular Data Engineering Weekly newsletter, shares his insights on building customer-facing data applications. He discusses the evolution of analytics infrastructure and the significance of technologies like data lineage and catalogs. Ananth delves into real-time data integration solutions, evaluating tools such as Druid and ClickHouse. He also offers guidance for software engineers looking to break into data roles, emphasizing the importance of SQL and a log-driven approach.
AI Snips
Chapters
Books
Transcript
Episode notes
Real-Time and Historical Data
- Combining historical and real-time data is a persistent challenge.
- Joining data in the database is generally preferred over joining in the stream due to potential optimizations.
Real-time System Challenges
- Real-time systems like Pinot and Druid, originating from ad-serving, are optimized for immutable events.
- SaaS applications require handling mutable data and object lifecycles, making Pinot's upsert functionality, while present, not ideal.
Data Formats
- Avoid JSON as your primary data format for data pipelines.
- Use a schema registry and leverage compilers for type checking during compilation.
