2min chapter

Software Engineering Radio - the podcast for professional software developers cover image

Episode 424: Sean Knapp on Dataflow Pipeline Automation

Software Engineering Radio - the podcast for professional software developers

CHAPTER

Is the DAG of All the Steps?

The traditional DAG describes the code that says perform each one of these incremental steps. The sort of new wave is really around declarative data pipelines. And to be declarative, you actually focus less on the steps of like extract from here, do this transform check the states and so on. All of that goes away and is the responsibility of the orchestration system in a declarative model with DAGs.

00:00

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.
App store bannerPlay store banner

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode