Data Engineering Podcast cover image

Data Engineering Podcast

Latest episodes

undefined
44 snips
Jun 29, 2025 • 54min

Enabling Agents In The Enterprise With A Platform Approach

Arun Joseph, an AI engineering leader and entrepreneur, discusses his journey in developing multi-agent systems. He emphasizes the transformative potential of agentic capabilities in businesses and shares insights on building robust data models and orchestration loops. Arun tackles the challenges of managing large-scale data contexts, the importance of unified context management to avoid silos, and the shift toward open-source platforms like LMOS. He also explores how these innovations can enhance decision-making and streamline enterprise data management.
undefined
33 snips
Jun 18, 2025 • 1h 2min

Dagster's New Era: Modularizing Data Transformation in the Age of AI

Nick Schrock, CTO and founder of Dagster Labs, shares his expertise on the transformational role of AI in data engineering. He discusses the importance of maintaining core data principles amid AI advancements and emphasizes the necessity of human oversight. Nick introduces Dagster's innovative components feature that modularizes data transformations, enhancing collaboration. The conversation also covers the balance between simplifying tools for non-technical users and maintaining customizability, addressing key challenges in the evolving data landscape.
undefined
7 snips
Jun 11, 2025 • 44min

AI and the Lakehouse: How Starburst is Pioneering New Workflows

Alex Albu, tech lead for AI initiatives at Starburst, dives into the fascinating world of integrating AI workloads with lakehouse architecture. He shares his journey from software engineering to championing AI enhancements at Starburst. The discussion covers innovative solutions like AI agents for data exploration and metadata enrichment. Alex addresses the hurdles of marrying AI with traditional data systems and reveals future visions for improved data formats and AI-driven tools, promising a revolution in data management.
undefined
53 snips
Jun 3, 2025 • 1h 1min

Amazon S3: The Backbone of Modern Data Systems

Mai-Lan Tomsen Bukovec, Vice President of Technology at AWS, reveals the remarkable evolution of Amazon S3, a key data repository since 2006. She shares fascinating insights on how S3 transformed from basic storage into the backbone of modern data architecture, enabling scalable data lakes. Discussion includes the importance of metadata, the integration of S3 with Iceberg, and innovations like strong consistency. Companies like Adobe and Netflix leverage S3 for efficiency, showcasing its role in navigating both structured and unstructured data challenges.
undefined
22 snips
May 29, 2025 • 42min

Scaling Data Operations With Platform Engineering

Chakravarti Kotaru, Director of a data platform at a leading online travel company with nearly two decades in scalable architectures, shares his expertise on data operations. He discusses the evolution from DevOps to platform engineering, emphasizing centralized management and automation with AWS Service Catalog. Kotaru elaborates on the challenges of database migrations, the integration of AI and ML for enhanced efficiency, and the vital role of organizational buy-in in successful data initiatives. A fascinating glimpse into modern data platform strategies!
undefined
43 snips
May 21, 2025 • 50min

From Data Discovery to AI: The Evolution of Semantic Layers

Shinji Kim, Founder and CEO of SelectStar, shares insights on the evolving role of semantic layers in AI. He discusses the journey from statistical analysis to data governance, highlighting challenges enterprises face with data access. The conversation covers the shift from centralized to decentralized data teams and the importance of metadata management. Shinji emphasizes the critical role of semantic modeling for business intelligence and how AI can enhance data accuracy. He also explores the future of semantic modeling in data warehouses, addressing operationalization challenges.
undefined
10 snips
May 13, 2025 • 46min

Balancing Off-the-Shelf and Custom Solutions in Data Engineering

Tulika Bhatt, a senior software engineer at Netflix specializing in impression data, shares her journey from BlackRock and Verizon to shaping data services at a top streaming service. She discusses the challenges of balancing off-the-shelf solutions with custom systems, utilizing technologies like Spark and Flink. Tulika dives into the intricacies of ensuring data quality and observability, emphasizing automation and robust alerting strategies. She also explores the integration of AI in data engineering, highlighting its potential and the hurdles faced in maximizing efficiency.
undefined
9 snips
May 5, 2025 • 60min

StarRocks: Bridging Lakehouse and OLAP for High-Performance Analytics

Sida Shen, a product manager at CelerData and a contributor to StarRocks, dives into the innovative world of high-performance analytical databases. He shares the origins of StarRocks, illustrating its evolution from Apache Doris into a robust Lakehouse query engine. Topics include handling high concurrency and low latency queries, bridging traditional OLAP with lakehouse architecture, and the importance of integration with formats like Apache Iceberg. Sida also emphasizes the challenges of denormalization and real-time data processing in modern analytics.
undefined
12 snips
Apr 28, 2025 • 1h 13min

Exploring NATS: A Multi-Paradigm Connectivity Layer for Distributed Applications

Derek Collison, the creator of NATS and CEO of Synadia, shares insights from his impressive background at Google and VMware. He discusses how NATS revolutionizes messaging systems with innovative features like the circuit breaker pattern and Jetstream. Derek highlights NATS’s advantages in edge computing, emphasizing its resilience and data persistence capabilities. He also addresses the challenges of open-source technology and shares thoughts on the future of connectivity in modern distributed systems, proving NATS's versatility across various industries.
undefined
37 snips
Apr 21, 2025 • 57min

Advanced Lakehouse Management With The LakeKeeper Iceberg REST Catalog

Victor Kessler, co-founder of Vakama and developer of Lakekeeper, dives into the world of advanced lakehouse management with a focus on Apache Iceberg. He discusses the pivotal role of metadata in data actionability and the evolution of data catalogs. Victor highlights innovative features of Lakekeeper, like integration with OpenFGA for access control and its deployment using Rust on Kubernetes. He also addresses the challenges of migrating data catalogs and the importance of community involvement in open-source projects for better data management.

The AI-powered Podcast Player

Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!
App store bannerPlay store banner
Get the app