Enhancing Data Accessibility and Governance with Gravitino

Sep 1, 2024

Junping Du, an expert in data management and the creator of Gravitino, discusses how this open-source metadata service revolutionizes data accessibility and governance. He explains Gravitino's unified interface for querying diverse data sources, addressing challenges in managing both structured and unstructured data. Junping highlights the importance of centralized governance and the tool's architectural design that promotes operational efficiency. Additionally, he talks about bridging gaps between data and AI professionals to foster collaboration and innovation in the field.

Ask episode

Chapters

Transcript

Episode notes

Intro

00:00 • 3min

Revolutionizing Data Access with Gravitino

02:37 • 20min

Navigating Unstructured Data Management and Governance

22:10 • 3min

Optimizing Data Management through Access Patterns and Tiered Storage

25:13 • 2min

Governance and Innovation with Gravitino

27:03 • 8min

Bridging Gaps in Data Management and AI Collaboration

35:20 • 3min

Summary
As data architectures become more elaborate and the number of applications of data increases, it becomes increasingly challenging to locate and access the underlying data. Gravitino was created to provide a single interface to locate and query your data. In this episode Junping Du explains how Gravitino works, the capabilities that it unlocks, and how it fits into your data platform.
Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Your host is Tobias Macey and today I'm interviewing Junping Du about Gravitino, an open source metadata service for a unified view of all of your schemas

Interview

Introduction
How did you get involved in the area of data management?
Can you describe what Gravitino is and the story behind it?
What problems are you solving with Gravitino?
- What are the methods that teams have relied on in the absence of Gravitino to address those use cases?
What led to the Hive Metastore being the default for so long?
- What are the opportunities for innovation and new functionality in the metadata service?
The documentation suggests that Gravitino has overlap with a number of tool categories such as table schema (Hive metastore), metadata repository (Open Metadata), data federation (Trino/Alluxio). What are the capabilities that it can completely replace, and which will require other systems for more comprehensive functionality?
What are the capabilities that you are explicitly keeping out of scope for Gravitino?
Can you describe the technical architecture of Gravitino?
- How have the design and scope evolved from when you first started working on it?
Can you describe how Gravitino integrates into an overall data platform?
- In a typical day, what are the different ways that a data engineer or data analyst might interact with Gravitino?
One of the features that you highlight is centralized permissions management. Can you describe the access control model that you use for unifying across underlying sources?
What are the most interesting, innovative, or unexpected ways that you have seen Gravitino used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Gravitino?
When is Gravitino the wrong choice?
What do you have planned for the future of Gravitino?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA