AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
The Data Stack Show
171: Machine Learning Pipelines Are Still Data Pipelines with Sandy Ryza of Dagster
Jan 3, 2024
Guest Sandy Ryza, an expert in machine learning pipelines, discusses the role of orchestrators in the lifecycle of data, changes in data ops and MLOps, data cleaning, and the overview of Dagster. They also explore the difference between data assets and tasks in data pipelines, defining lineage and data assets in Dagster, and the benefits of a unified orchestration framework. Additionally, they touch on orchestration in the development phase and the emergence of the analytics engineer role.
55:50
Episode guests
AI Summary
AI Chapters
Episode notes
Podcast summary created with Snipd AI
Quick takeaways
- The boundaries between data engineering, ML engineering, and data science roles are becoming increasingly blurred, allowing individuals to explore different areas and follow their curiosity without needing a complete career change.
- Orchestrators like Dagster play a crucial role in managing and executing data pipelines for both analytics and ML workloads, providing a flexible execution substrate for experimentation and reliability across the entire development lifecycle.
Deep dives
The Blurring Lines in Data Roles
The boundaries between data engineering, ML engineering, and data science roles are becoming increasingly blurred. In the past, different roles had distinct responsibilities, but now there is more overlap and fluidity. Proficiency in data modeling and infrastructure are key aspects of these roles. The tooling available now allows for collaboration and crossover between Python and SQL, providing flexibility for individuals to explore different areas and follow their curiosity. This blurring of lines is sparking creativity and enabling individuals to pursue their interests without needing a complete career change.
Get the Snipd
podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any
moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share
& Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode