Data Engineering Podcast

Petr Janda, CEO of Synq, discusses the importance of data reliability and transparency, emphasizing treating data systems like engineering systems. Synq's platform helps manage incidents, data dependencies, and ensures data quality. By integrating data into business processes, Synq empowers data teams to drive meaningful change and optimize data management.

Dipti Borkar, an expert at Microsoft Fabric, discusses accelerating enterprise adoption of data lakehouse architectures. She shares experiences and use cases for the Fabric service, highlighting its integration with Spark engine and X-Table for seamless interop. The episode explores optimizing Fabric for enterprise use, innovations in data lake analytics, and the future projections in AI role in data engineering.

Learn how Stripe utilizes Trino and Iceberg for their data lakehouse, including insights on business analytics, challenges with large datasets, optimizing with Iceberg, and transitioning to REST catalog. Discover the advantages of monitoring queries and managing multi-tool ecosystems with Trino and Spark. Explore the challenges and innovations in cloud data management with Trino and Iceberg at Stripe.

Dive into the world of Flink stream processing with Ronen Korman and Stav Elkayam from Datorios. They discuss how observability can enhance visibility into Flink internals, address challenges in real-time data processing, explore the role of Flink in AI applications, and highlight the evolution and integration of Datorios with Apache Flink for stream processing.

Nicola Askham, accidental data governance expert, discusses practical steps for implementing data governance. Topics include benefits of data governance, pitfalls to avoid, securing executive support, engaging stakeholders, overcoming roadblocks, navigating shadow IT, AI integration, and challenges faced by organizations in data governance implementation.

Experienced data engineer Sriram Panyam shares insights on managing data migration projects in high traffic environments. Topics include strategies for maintaining data consistency, pitfalls to avoid, and involving application teams for organizational alignment. The episode explores challenges in data management and the importance of architectural soundness.

Zenlytic is revolutionizing business intelligence systems by using AI agents that allow users to converse with their data. The podcast delves into the challenges and advancements in generative AI, highlighting the difference between AI chatbots and AI agents. The team discusses the importance of fundamental knowledge in AI models, navigating data lake complexity, and scalability considerations for B2B applications. They also explore the evolving role of AI agents in enhancing text data analysis and business intelligence.

Explore the challenges of release management for data platform services and logic, including complexities of testing data pipelines, strategies for data integrity testing, development environment challenges in Daxter pipelines, and the evolution of validation and release management in data systems.

Peter Voss, a pioneer in cognitive AI, discusses the shift towards human-like intelligence in AI, emphasizing learning over statistical prediction. The podcast explores the evolution from narrow AI to AGI, contrasts generative systems with cognitive AI, and highlights the challenges and benefits of achieving human-level AGI. Voss advocates for maximizing AI capabilities, leveraging open-source resources, and prioritizing transparency and explainability in AI models.

Tsavo Knott, creator of Pieces, discusses simplifying AI integration into developer workflows with a powerful collection of tools. He explains data collection, model types, and incorporating Pieces as a second brain. The podcast explores the impact of AI on developer tooling, personalized AI tools, challenges in machine learning, building integrated systems, and enhancing developer workflows with the Pieces tool.