Data Engineering Podcast An Exploration Of The Impediments To Reusable Data Pipelines
70 snips
Dec 8, 2024 Max Beauchemin, a data engineer with two decades of experience and founder of Preset, dives into the complexities of reusable data pipelines. He discusses the "write everything twice" problem, emphasizing the need for collaboration and shared reference implementations. Max explores the challenges of managing diverse SQL dialects and the evolving role of data engineers, likening it to front-end development. He envisions generative AI aiding knowledge distribution and encourages the community to engage in sharing templates to drive innovation in the field.
AI Snips
Chapters
Transcript
Episode notes
Lack of Pipeline Reuse
- Data engineers often rewrite similar pipelines across organizations.
- This lack of code reuse is a significant inefficiency, especially for transformation tasks.
Data Integration Progress
- The data integration layer (the "EL" of ELT) is becoming more mature, with tools like Airbyte and Fivetran.
- This progress is a necessary foundation for reusable transformations (the "T"), but more work is needed.
DBT Reusability Challenges
- Reusable DBT projects are limited by SQL dialect differences and messy parameterized pipelines.
- Managing various SQL dialects and dynamically generating pipelines are key challenges.