The Data Stack Show 246: AI, Abstractions, and the Future of Data Engineering with Pete Hunt of Dagster
28 snips
May 28, 2025 Pete Hunt, CEO of Dagster and ex-Facebook engineer known for his work on React.js, dives into the evolution of data practices and the impact of generative AI on data engineering. He discusses the challenges of integrating acquired companies and the importance of abstraction in managing complexity. The conversation highlights the role of collaboration, trust in AI, and the necessity of guardrails within workflows. Pete also shares valuable insights into fostering empathy in tech and how innovation can shape the future of data integration.
AI Snips
Chapters
Transcript
Episode notes
Early Data Query Challenges
- Pete Hunt shared his experience using Hive at Facebook, where queries took 20 minutes, causing delays and wrong answers.
- The transition to Presto enabled much faster queries, dramatically improving data-driven decision making.
Trust and Safety Startup Experience
- Pete Hunt founded Smite, a trust and safety company focusing on real-time detection of fake accounts.
- He learned adversarial data problems require a mix of heuristics, anomaly detection, and machine learning with low latency responses.
Big Complexity in Modern Data Stack
- The modern data stack solved big data speed with tools like BigQuery and Snowflake.
- This led to an era of big complexity requiring software engineering best practices for manageability.