Contributor

Big Complexity Revisited: Dagster with Pete Hunt

10 snips
Jun 4, 2025
Pete Hunt, CEO of Dagster Labs and former React.js engineer at Facebook, shares insights on Dagster, an innovative open-source data orchestration platform. He discusses the evolution from traditional to low-code solutions, highlighting software-defined data assets. The conversation touches on AI's evolving role in data pipelines and the importance of observability tools in ensuring data quality. Hunt emphasizes treating data processing with engineering discipline to navigate complexities and improve productivity across organizations.
Ask episode
AI Snips
Chapters
Transcript
Episode notes
INSIGHT

Dagster's Core Purpose

  • Dagster is an open source data orchestration platform for building and managing data pipelines efficiently.
  • It helps operate, understand, optimize, and maintain complex data workflows over time.
ANECDOTE

Pete Hunt's CEO Journey

  • Pete Hunt joined Dagster as head of engineering before becoming CEO after about a year.
  • Nick Schrock wanted to focus on creative CTO work while Pete took over CEO operational duties.
INSIGHT

Big Complexity in Data Platforms

  • Modern data platforms face 'big complexity' managing large, heterogeneous data stacks and data freshness problems.
  • Dagster treats data orchestrations as software engineering projects with observability and deep understanding, unlike black-box workflow engines.
Get the Snipd Podcast app to discover more snips from this episode
Get the app