Prodity: Product by Design
Lessons in Data Engineering: Scaling, AI, and Open Source with Sandy Ryza
Feb 7, 2025
Sandy Ryza, a lead engineer on Dagster, shares his rich journey from software engineering to data science. He dives into the evolution of data engineering, emphasizing its increasing complexity and the vital role of AI in shaping data platforms. Sandy discusses best practices for managing data, highlighting the integration of software engineering principles. He also reflects on the future of open-source tools and the importance of data ownership in modern infrastructures. His insights offer great value for both seasoned professionals and newcomers in the field.
46:28
Episode guests
AI Summary
AI Chapters
Episode notes
Podcast summary created with Snipd AI
Quick takeaways
- Sandy Ryza emphasizes the necessity of software engineering principles to manage increasing complexity in modern data pipelines and ensure scalable data platforms.
- The rise of unstructured data and AI will reshape data engineering, making intentional platform design and interoperability critical for future success.
Deep dives
Sandy's Journey in Data Engineering
Sandy shares his multifaceted career journey that has revolved around data engineering, starting as a software engineer building tools for complex data sets. After transitioning to a data practitioner role, he faced various challenges that fueled his desire to refocus on creating better tools. This culminated in his current work on Dagster, an orchestration and data management tool aimed at improving data pipelines. His experiences reflect a holistic understanding of both the technical and practical aspects of data engineering, positioning him as a thought leader in the field.
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
