From Academia to Industry: Bridging Data Engineering Challenges

16 snips

Aug 26, 2025

In this engaging discussion, Professor Paul Groth from the University of Amsterdam shares his expertise in AI systems and intelligent data engineering. He dives into the evolution of data provenance and lineage, illustrating its significance in today's workflows. Paul also highlights the transformative impact of large language models on knowledge graph construction and data integration. The conversation addresses the synergy between academia and industry, emphasizing human-AI collaboration and the need for tailored data management solutions.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

ANECDOTE

Journey From AI To Data Integration

Paul Groth started in AI then shifted to distributed computing and data provenance during his PhD.
He later built early graph databases and a large biomedical knowledge graph called OpenFax integrating ~20 sources.

INSIGHT

Lineage Expanded Beyond Databases

Paul treats data provenance and lineage as largely the same but notes the field broadened from DB internals to cross-system workflow tracing.
Modern interest focuses on tracing results across the organization, not just within one database.

ADVICE

Automate Graphs And Test Data Quality

Automate knowledge graph construction and mappings when possible to lower integration cost.
Invest in data quality and measure how data prep choices affect downstream ML models.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Summary
In this episode of the Data Engineering Podcast Professor Paul Groth, from the University of Amsterdam, talks about his research on knowledge graphs and data engineering. Paul shares his background in AI and data management, discussing the evolution of data provenance and lineage, as well as the challenges of data integration. He explores the impact of large language models (LLMs) on data engineering, highlighting their potential to simplify knowledge graph construction and enhance data integration. The conversation covers the evolving landscape of data architectures, managing semantics and access control, and the interplay between industry and academia in advancing data engineering practices, with Paul also sharing insights into his work with the intelligent data engineering lab and the importance of human-AI collaboration in data engineering pipelines.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
Your host is Tobias Macey and today I'm interviewing Paul Groth about his research on knowledge graphs and data engineering

Interview

Introduction
How did you get involved in the area of data management?
Can you start by describing the focus and scope of your academic efforts?
Given your focus on data management for machine learning as part of the INDELab, what are some of the developing trends that practitioners should be aware of?
- ML architectures / systems changing (matteo interlandi) GPUs for data mangement
You have spent a large portion of your career working with knowledge graphs, which have largely been a niche area until recently. What are some of the notable changes in the knowledge graph ecosystem that have resulted from the introduction of LLMs?
What are some of the other ways that you are seeing LLMs change the methods of data engineering?
- There are numerous vague and anecdotal references to the power of LLMs to unlock value from unstructured data. What are some of the realitites that you are seeing in your research?
A majority of the conversations in this podcast are focused on data engineering in the context of a business organization. What are some of the ways that management of research data is disjoint from the methods and constraints that are present in business contexts?
What are the most interesting, innovative, or unexpected ways that you have seen LLM used in data management?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on data engineering research?
What do you have planned for the future of your research in the context of data engineering, knowledge graphs, and AI?

Contact Info

Website
email

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA