Data Engineering Podcast Unpacking The Seven Principles Of Modern Data Pipelines
47:03
Modern Data Pipeline Definition
- Modern data pipelines handle many diverse sources and larger volumes with near real-time requirements.
- They use cloud-first, scalable infrastructure and support multiple storage layers including lakes and warehouses.
Seven Principles to Modernize Pipelines
- Use cloud-first, zero infrastructure management to quickly set up scalable data pipelines.
- Favor ELT over ETL and enable configuration with common languages like SQL and Python for faster delivery.
Marketing Data Use Cases Influence
- Marketing data use cases revealed the need to rapidly connect to many SaaS apps, even niche ones.
- Digital agencies required multi-tenant pipelines to serve many clients with similar data sources.
Get the Snipd Podcast app to discover more snips from this episode
Get the app 1 chevron_right 2 chevron_right 3 chevron_right 4 chevron_right 5 chevron_right 6 chevron_right 7 chevron_right 8 chevron_right 9 chevron_right 10 chevron_right 11 chevron_right 12 chevron_right 13 chevron_right 14 chevron_right
Introduction
00:00 • 3min
The Seven Principles of Modern Data Pipelines
02:56 • 4min
The Seven Principles of a Modern Data Pipeline
06:43 • 3min
The Importance of a Modern Data Pipeline
09:27 • 3min
The Different Use Cases for Data Engineering
12:55 • 6min
The Importance of Data Ability to Deliver Faster
18:38 • 4min
The Core Requirements for a Pipeline Infrastructure to Support the Principles of the Modern Workflow
22:21 • 4min
The Challenges of Implementing No Code in Data Teams
26:01 • 3min
The Impact of Tool Selection on Data Pipelines
28:35 • 3min
The Impact of Managed Services on Data Use and Adoption
31:08 • 4min
The Seven Principles of Data Technology
34:45 • 3min
The Evolution of Hybrid Data Warehouses
37:19 • 3min
The Future of Generative AI
40:00 • 2min
The Seven Principles of Data Management
42:15 • 5min
Summary
Data pipelines are the core of every data product, ML model, and business intelligence dashboard. If you're not careful you will end up spending all of your time on maintenance and fire-fighting. The folks at Rivery distilled the seven principles of modern data pipelines that will help you stay out of trouble and be productive with your data. In this episode Ariel Pohoryles explains what they are and how they work together to increase your chances of success.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack
- This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold
- Your host is Tobias Macey and today I'm interviewing Ariel Pohoryles about the seven principles of modern data pipelines
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you start by defining what you mean by a "modern" data pipeline?
- At Rivery you published a white paper identifying seven principles of modern data pipelines:
- Zero infrastructure management
- ELT-first mindset
- Speaks SQL and Python
- Dynamic multi-storage layers
- Reverse ETL & operational analytics
- Full transparency
- Faster time to value
- What are the applications of data that you focused on while identifying these principles?
- How do the application of these principles influence the ability of organizations and their data teams to encourage and keep pace with the use of data in the business?
- What are the technical components of a pipeline infrastructure that are necessary to support a "modern" workflow?
- How do the technologies involved impact the organizational involvement with how data is applied throughout the business?
- When using managed services, what are the ways that the pricing model acts to encourage/discourage experimentation/exploration with data?
- What are the most interesting, innovative, or unexpected ways that you have seen these seven principles implemented/applied?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working with customers to adapt to these principles?
- What are the cases where some/all of these principles are undesirable/impractical to implement?
- What are the opportunities for further advancement/sophistication in the ways that teams work with and gain value from data?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
Links
- Rivery
- 7 Principles Of The Modern Data Pipeline
- ELT
- Reverse ETL
- Martech Landscape
- Data Lakehouse
- Databricks
- Snowflake
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Sponsored By:
- Datafold:  This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting [dataengineeringpodcast.com/datafold](https://www.dataengineeringpodcast.com/datafold) today!
- Rudderstack:  Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack)