Data Engineering Podcast

Kate Shaw, Senior Product Manager for Data at SnapLogic, dives into the complexities of legacy systems and their modern replacements. She highlights that legacy isn't just age—it's about risk and innovation barriers. They discuss technical debt, lost context from turnover, and the dangers of 'if it ain’t broke.' Shaw advocates for composable architectures and planning exit strategies from day one. Additionally, she touches on integrating legacy systems into AI initiatives and the importance of transparency in data governance. A must-listen for anyone navigating modernization!

Nick Schrock, CTO and founder of Dagster Labs, shares his insights on agentic analytics and the innovative Compass tool he developed. He explains how Compass transforms data teams into stewards of context while integrating seamlessly with Slack for enhanced collaboration. Schrock discusses the implications of agentic systems on Conway's Law and the need for new infrastructure to support these workflows. He also highlights cost control strategies and the future of context engineering in software development, unveiling his optimistic outlook on AI advancements.

Vijay Subramanian, CEO of Trace and former data leader at Rent the Runway, dives into the revolutionary concept of metric trees as a data model that mirrors a company's business framework. He reveals how traditional dashboards often miss the mark and how metric trees can enhance analytical workflows by clarifying cause and effect. Vijay shares insights on leveraging these trees alongside AI agents for operational analytics and discusses real-world applications like modeling customer journeys. He also emphasizes the importance of collaboration with business teams to effectively implement this innovative approach.

Brijesh Tripathi, CEO of Flex AI, combines his rich background in AI and HPC architecture to revolutionize AI infrastructure. He discusses the burdens of DevOps that slow down small AI teams and highlights Flex AI's innovative workload-as-a-service approach. Brijesh breaks down the challenges of accessing heterogeneous compute, the importance of consistent Kubernetes layers, and how to smooth costs for spiky workloads. He also shares insights on handling real-time vs. best-effort workloads, maximizing utilization, and ensuring that AI teams can focus on creativity instead of complexity.

Mark Brooker, VP and Distinguished Engineer at AWS, dives into how agentic workflows are revolutionizing database infrastructure. He shares insights on why agents demand serverless, elastic databases and discusses the shift from traditional data models to vectors and relational databases. Mark explores the significance of tools like D-SQL for managing global agent workloads and highlights real-world applications, such as agent-driven SQL fuzzing. He also emphasizes the need for improved identity and authorization in our evolving data landscape.

Hannes Mühleisen and Mark Raasveldt, key figures behind DuckDB, dive into their latest project, Duck Lake, aiming to simplify the lakehouse ecosystem. They discuss how Duck Lake stands out with its unified SQL database, making metadata management a breeze. The duo shares their vision for decentralized processing, local-first data architecture, and benefits like data inlining and encryption. They also touch on its seamless integration with existing systems, showcasing how it can transform data workflows and enhance user experiences.

Serge Gershkovich, Head of Product at SQL DBM and a Snowflake data expert, dives into the socio-technical aspects of data modeling. He emphasizes that effective data modeling is crucial for aligning business needs with technical structures, debunking myths about its importance. The discussion explores challenges in complex environments and the evolving role of AI in data management. Serge advocates for collaboration between business teams and data professionals, highlighting how clear communication can enhance trust and mitigate issues related to data quality.

In this engaging discussion, Professor Paul Groth from the University of Amsterdam shares his expertise in AI systems and intelligent data engineering. He dives into the evolution of data provenance and lineage, illustrating its significance in today's workflows. Paul also highlights the transformative impact of large language models on knowledge graph construction and data integration. The conversation addresses the synergy between academia and industry, emphasizing human-AI collaboration and the need for tailored data management solutions.

Prashanth Rao, an AI engineer at KuzuDB, delves into the cutting-edge features of their embeddable graph database. He explains how KuzuDB tackles performance issues with innovative columnar storage and unique join algorithms. The conversation reveals KuzuDB's potential for enhancing graph applications, especially in edge computing and ephemeral workloads. Prashanth also discusses the growing interest in graph databases for AI integration and how Kuzu can seamlessly work with other data formats like Iceberg and Parquet.

Lucas Thelosen and Drew Gilson, co-founders of Gravity, delve into the transformative impact of AI in data analytics. They discuss their creation of Orion, an autonomous data analyst designed to bridge data and decision-making. The conversation highlights how AI democratizes access to data insights for businesses of all sizes, allowing data analysts to focus on strategic tasks. They also emphasize the importance of accuracy and trustworthiness in AI-driven workflows, sharing insights on how companies can cultivate a data-driven culture.