Data Engineering Podcast

An Exploration Of The Impediments To Reusable Data Pipelines

Dec 8, 2024

Max Beauchemin, a data engineer with two decades of experience and founder of Preset, dives into the complexities of reusable data pipelines. He discusses the "write everything twice" problem, emphasizing the need for collaboration and shared reference implementations. Max explores the challenges of managing diverse SQL dialects and the evolving role of data engineers, likening it to front-end development. He envisions generative AI aiding knowledge distribution and encourages the community to engage in sharing templates to drive innovation in the field.

51:32

Creator website

Episode guests

Maxime Beauchemin

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Code reuse in data engineering is hindered by the lack of standardization and tooling, leading to inefficient practices across different organizations.

The rise of generative AI and better open-source collaboration could greatly enhance documentation, sharing, and ultimately the reusability of data pipelines.

Deep dives

The Challenge of Code Reusability in Data Engineering

Code reuse in data engineering remains an elusive goal, as engineers often find themselves rewriting similar data pipelines in different organizations. Despite the expectations following the open-sourcing of tools like Apache Airflow, significant barriers persist, including limitations in tooling, ecosystem, and education. The conversation highlights the repetitiveness of tasks, particularly in data transformation, where engineers frequently reimplement SQL code without any standardization across organizations, leading to inefficiency. A key point raised is the need for more accessible frameworks and reference implementations that could enable greater code sharing and inspire data engineers to take advantage of shared knowledge.

Intro

3min

Challenges in Code Reusability for Data Pipelines

4min

Navigating Data Engineering Complexities

20min

Navigating Data Engineering Challenges

17min

Harnessing Generative AI in Data Engineering

5min

Building a Community of Knowledge in Data Engineering

3min

Summary
In this episode of the Data Engineering Podcast the inimitable Max Beauchemin talks about reusability in data pipelines. The conversation explores the "write everything twice" problem, where similar pipelines are built without code reuse, and discusses the challenges of managing different SQL dialects and relational databases. Max also touches on the evolving role of data engineers, drawing parallels with front-end engineering, and suggests that generative AI could facilitate knowledge capture and distribution in data engineering. He encourages the community to share reference implementations and templates to foster collaboration and innovation, and expresses hopes for a future where code reuse becomes more prevalent.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
Your host is Tobias Macey and today I'm joined again by Max Beauchemin to talk about the challenges of reusability in data pipelines

Interview

Introduction
How did you get involved in the area of data management?
Can you start by sharing your current thesis on the opportunities and shortcomings of code and component reusability in the data context?
- What are some ways that you think about what constitutes a "component" in this context?
The data ecosystem has arguably grown more varied and nuanced in recent years. At the same time, the number and maturity of tools has grown. What is your view on the current trend in productivity for data teams and practitioners?
What do you see as the core impediments to building more reusable and general-purpose solutions in data engineering?
- How can we balance the actual needs of data consumers against their requests (whether well- or un-informed) to help increase our ability to better design our workflows for reuse?
In data engineering there are two broad approaches; code-focused or SQL-focused pipelines. In principle one would think that code-focused environments would have better composability. What are you seeing as the realities in your personal experience and what you hear from other teams?
When it comes to SQL dialects, dbt offers the option of Jinja macros, whereas SDF and SQLMesh offer automatic translation. There are also tools like PRQL and Malloy that aim to abstract away the underlying SQL. What are the tradeoffs across those options that help or hinder the portability of transformation logic?
Which layers of the data stack/steps in the data journey do you see the greatest opportunity for improving the creation of more broadly usable abstractions/reusable elements?
low/no code systems for code reuse
impact of LLMs on reusability/composition
impact of background on industry practices (e.g. DBAs, sysadmins, analysts vs. SWE, etc.)
polymorphic data models (e.g. activity schema)
What are the most interesting, innovative, or unexpected ways that you have seen teams address composability and reusability of data components?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on data-oriented tools and utilities?
What are your hopes and predictions for sharing of code and logic in the future of data engineering?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.

Links