Software Engineering Radio - the podcast for professional software developers cover image

Episode 424: Sean Knapp on Dataflow Pipeline Automation

Software Engineering Radio - the podcast for professional software developers

00:00

Is the DAG of All the Steps?

The traditional DAG describes the code that says perform each one of these incremental steps. The sort of new wave is really around declarative data pipelines. And to be declarative, you actually focus less on the steps of like extract from here, do this transform check the states and so on. All of that goes away and is the responsibility of the orchestration system in a declarative model with DAGs.

Transcript
Play full episode

The AI-powered Podcast Player

Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!
App store bannerPlay store banner
Get the app