Contributor Big Complexity Revisited: Dagster with Pete Hunt
20 snips
Jun 4, 2025 Pete Hunt, CEO of Dagster Labs and former React.js engineer at Facebook, shares insights on Dagster, an innovative open-source data orchestration platform. He discusses the evolution from traditional to low-code solutions, highlighting software-defined data assets. The conversation touches on AI's evolving role in data pipelines and the importance of observability tools in ensuring data quality. Hunt emphasizes treating data processing with engineering discipline to navigate complexities and improve productivity across organizations.
AI Snips
Chapters
Transcript
Episode notes
Dagster's Core Purpose
- Dagster is an open source data orchestration platform for building and managing data pipelines efficiently.
- It helps operate, understand, optimize, and maintain complex data workflows over time.
Pete Hunt's CEO Journey
- Pete Hunt joined Dagster as head of engineering before becoming CEO after about a year.
- Nick Schrock wanted to focus on creative CTO work while Pete took over CEO operational duties.
Big Complexity in Data Platforms
- Modern data platforms face 'big complexity' managing large, heterogeneous data stacks and data freshness problems.
- Dagster treats data orchestrations as software engineering projects with observability and deep understanding, unlike black-box workflow engines.