Data Engineering Podcast Re-Bundling The Data Stack With Data Orchestration And Software Defined Assets Using Dagster
58:14
Orchestration Lost Visibility — Assets Restore It
- Orchestration layers are losing visibility because constituent tools now own the internal DAGs instead of the orchestrator.
- Software-defined assets re-map orchestration to asset-level, restoring a single place to understand platform lineage.
Distribute Complexity With Asset-Centric Design
- Use software-defined assets to distribute complexity across teams and avoid a single centralized DAG file.
- Deploy assets from independent repos and stitch them together so teams can work autonomously yet remain visible.
Customer Wrote Feast Integration To Unify Lineage
- A cloud customer integrated Dagster, dbt, and Feast by writing a custom integration to expose Feast as an asset.
- That integration united data engineering and ML lineage into one coherent graph.
Get the Snipd Podcast app to discover more snips from this episode
Get the app 1 chevron_right 2 chevron_right 3 chevron_right 4 chevron_right 5 chevron_right 6 chevron_right 7 chevron_right 8 chevron_right 9 chevron_right 10 chevron_right 11 chevron_right 12 chevron_right 13 chevron_right 14 chevron_right 15 chevron_right 16 chevron_right 17 chevron_right 18 chevron_right 19 chevron_right 20 chevron_right 21 chevron_right 22 chevron_right 23 chevron_right
Introduction
00:00 • 2min
Softward Is Defined Assets - What's the Motivation?
02:08 • 3min
Python Data Platforms - What Are the Outcomes?
05:27 • 2min
Using Github to Find Assets in a Data Platform
07:16 • 3min
The Core of a Sovereignfined Asset
10:16 • 2min
The SopDefined Acid Layer
12:12 • 2min
The Journey of Sopra Defined Assets
14:31 • 3min
The Biggest Lesson You've Learned From Daxter Day
17:50 • 2min
The Stable Corps Foundation That We're Building Everything On
19:44 • 2min
Cloud G a Release - What's Coming Out?
21:46 • 2min
Data Engineering
24:09 • 2min
Using the Orchestrator to Easily Deploy Test Environments?
26:38 • 2min
Cloud Computing and the Datalake Use Case - What's Next?
28:48 • 2min
Daxter Day
31:11 • 3min
Data Day
33:42 • 2min
Dexer Cloud Platform - What Are Some of the New and Upcoming Capabilities?
36:10 • 3min
Open Source vs Protearial Platform - Application Complexity, Operation and Enterprise
38:59 • 3min
Managing Operational Complexity
41:29 • 2min
The Open Source Cloud Platform - What's the Process Behind That?
43:55 • 3min
The Ondato Release - What You've Learned?
47:25 • 2min
Cloud and Daxter Cloud
49:11 • 3min
Airflows vs Daxter - Which Is Better?
52:34 • 2min
What's the Biggest Gap in Data Management?
54:08 • 4min
Summary
The current stage of evolution in the data management ecosystem has resulted in domain and use case specific orchestration capabilities being incorporated into various tools. This complicates the work involved in making end-to-end workflows visible and integrated. Dagster has invested in bringing insights about external tools’ dependency graphs into one place through its "software defined assets" functionality. In this episode Nick Schrock discusses the importance of orchestration and a central location for managing data systems, the road to Dagster’s 1.0 release, and the new features coming with Dagster Cloud’s general availability.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
- RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.
- Data teams are increasingly under pressure to deliver. According to a recent survey by Ascend.io, 95% in fact reported being at or over capacity. With 72% of data experts reporting demands on their team going up faster than they can hire, it’s no surprise they are increasingly turning to automation. In fact, while only 3.5% report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. 85%!!! That’s where our friends at Ascend.io come in. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Go to dataengineeringpodcast.com/ascend and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.
- Your host is Tobias Macey and today I’m interviewing Nick Schrock about software defined assets and improving the developer experience for data orchestration with Dagster
Interview
- Introduction
- How did you get involved in the area of data management?
- What are the notable updates in Dagster since the last time we spoke? (November, 2021)
- One of the core concepts that you introduced and then stabilized in recent releases is the "software defined asset" (SDA). How have your users reacted to this capability?
- What are the notable outcomes in development and product practices that you have seen as a result?
- What are the changes to the interfaces and internals of Dagster that were necessary to support SDA?
- How did the API design shift from the initial implementation once the community started providing feedback?
- You’re releasing the stable 1.0 version of Dagster as part of something called "Dagster Day" on August 9th. What do you have planned for that event and what does the release mean for users who have been refraining from using the framework until now?
- Along with your 1.0 commitment to a stable interface in the framework you are also opening your cloud platform for general availability. What are the major lessons that you and your team learned in the beta period?
- What new capabilities are coming with the GA release?
- A core thesis in your work on Dagster is that developer tooling for data professionals has been lacking. What are your thoughts on the overall progress that has been made as an industry?
- What are the sharp edges that still need to be addressed?
- A core facet of product-focused software development over the past decade+ is CI/CD and the use of pre-production environments for testing changes, which is still a challenging aspect of data-focused engineering. How are you thinking about those capabilities for orchestration workflows in the Dagster context?
- What are the missing pieces in the broader ecosystem that make this a challenge even with support from tools and frameworks?
- How has the situation improved in the recent past and looking toward the near future?
- What role does the SDA approach have in pushing on these capabilities?
- What are the most interesting, innovative, or unexpected ways that you have seen Dagster used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on bringing Dagster to 1.0 and cloud to GA?
- When is Dagster/Dagster Cloud the wrong choice?
- What do you have planned for the future of Dagster and Elementl?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don’t forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
Links
- Dagster Day
- Dagster
- Elementl
- GraphQL
- Unbundling Airflow
- Feast
- Spark SQL
- Dagster Cloud Branch Deployments
- Dagster custom I/O manager
- LakeFS
- Iceberg
- Project Nessie
- Prefect
- Astronomer
- Temporal
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA