From Academia to Industry: Bridging Data Engineering Challenges
Data Engineering Podcast
00:00
Understanding Data Provenance and Lineage in Modern Workflows
This chapter explores data provenance and data lineage, distinguishing the two concepts while discussing their importance in data engineering. It emphasizes the shift from traditional database transformations to comprehensive tracking of workflows across various systems to help organizations trace data origins.
Transcript
Play full episode
Transcript
Episode notes
Summary
In this episode of the Data Engineering Podcast Professor Paul Groth, from the University of Amsterdam, talks about his research on knowledge graphs and data engineering. Paul shares his background in AI and data management, discussing the evolution of data provenance and lineage, as well as the challenges of data integration. He explores the impact of large language models (LLMs) on data engineering, highlighting their potential to simplify knowledge graph construction and enhance data integration. The conversation covers the evolving landscape of data architectures, managing semantics and access control, and the interplay between industry and academia in advancing data engineering practices, with Paul also sharing insights into his work with the intelligent data engineering lab and the importance of human-AI collaboration in data engineering pipelines.
Announcements
In this episode of the Data Engineering Podcast Professor Paul Groth, from the University of Amsterdam, talks about his research on knowledge graphs and data engineering. Paul shares his background in AI and data management, discussing the evolution of data provenance and lineage, as well as the challenges of data integration. He explores the impact of large language models (LLMs) on data engineering, highlighting their potential to simplify knowledge graph construction and enhance data integration. The conversation covers the evolving landscape of data architectures, managing semantics and access control, and the interplay between industry and academia in advancing data engineering practices, with Paul also sharing insights into his work with the intelligent data engineering lab and the importance of human-AI collaboration in data engineering pipelines.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
- Your host is Tobias Macey and today I'm interviewing Paul Groth about his research on knowledge graphs and data engineering
- Introduction
- How did you get involved in the area of data management?
- Can you start by describing the focus and scope of your academic efforts?
- Given your focus on data management for machine learning as part of the INDELab, what are some of the developing trends that practitioners should be aware of?
- ML architectures / systems changing (matteo interlandi) GPUs for data mangement
- You have spent a large portion of your career working with knowledge graphs, which have largely been a niche area until recently. What are some of the notable changes in the knowledge graph ecosystem that have resulted from the introduction of LLMs?
- What are some of the other ways that you are seeing LLMs change the methods of data engineering?
- There are numerous vague and anecdotal references to the power of LLMs to unlock value from unstructured data. What are some of the realitites that you are seeing in your research?
- A majority of the conversations in this podcast are focused on data engineering in the context of a business organization. What are some of the ways that management of research data is disjoint from the methods and constraints that are present in business contexts?
- What are the most interesting, innovative, or unexpected ways that you have seen LLM used in data management?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on data engineering research?
- What do you have planned for the future of your research in the context of data engineering, knowledge graphs, and AI?
- Website
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.
- INDELab
- Data Provenance
- Elsevier
- SIGMOD 2025
- Digital Twin
- Knowledge Graph
- WikiData
- KuzuDB
- data.world
- GraphRAG
- SPARQL
- Semantic Web
- GQL == Graph Query Language
- Cypher
- Amazon Neptune
- RDF == Resource Description Framework
- SwellDB
- FlockMTL
- DuckDB
- Matteo Interlandi
- Paolo Papotti
- Neuromorphic Computing
- Point Clouds
- Longform.ai
- BASIL DB
The AI-powered Podcast Player
Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!
