Data Engineering Podcast

Enhancing Data Accessibility and Governance with Gravitino

Sep 1, 2024

Junping Du, an expert in data management and the creator of Gravitino, discusses how this open-source metadata service revolutionizes data accessibility and governance. He explains Gravitino's unified interface for querying diverse data sources, addressing challenges in managing both structured and unstructured data. Junping highlights the importance of centralized governance and the tool's architectural design that promotes operational efficiency. Additionally, he talks about bridging gaps between data and AI professionals to foster collaboration and innovation in the field.

38:41

Creator website

Episode guests

Junping Du

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Gravitino provides a unified interface to access diverse data types, enhancing data accessibility and optimizing workflows in modern AI applications.

By standardizing permissions and simplifying access control, Gravitino improves data governance and organizational efficiency while mitigating access-related risks.

Deep dives

Unifying Data Catalogs

Gravitino aims to break down data silos created by diverse data lakes and cloud vendors, achieving a unified view of schemas through its open-source metadata service. It has evolved from insights gathered over years of working with Hadoop technology, discovering that various systems often isolate unstructured and structured data. By establishing a metadata lake, Gravitino enables seamless access to both types of data, allowing teams to manage and monitor AI workloads effectively. This capability empowers data engineers to conduct better data quality monitoring and make informed decisions regarding data lifecycle management.

Intro

3min

Revolutionizing Data Access with Gravitino

20min

Navigating Unstructured Data Management and Governance

3min

Optimizing Data Management through Access Patterns and Tiered Storage

2min

Governance and Innovation with Gravitino

8min

Bridging Gaps in Data Management and AI Collaboration

3min

Summary
As data architectures become more elaborate and the number of applications of data increases, it becomes increasingly challenging to locate and access the underlying data. Gravitino was created to provide a single interface to locate and query your data. In this episode Junping Du explains how Gravitino works, the capabilities that it unlocks, and how it fits into your data platform.
Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Your host is Tobias Macey and today I'm interviewing Junping Du about Gravitino, an open source metadata service for a unified view of all of your schemas

Interview

Introduction
How did you get involved in the area of data management?
Can you describe what Gravitino is and the story behind it?
What problems are you solving with Gravitino?
- What are the methods that teams have relied on in the absence of Gravitino to address those use cases?
What led to the Hive Metastore being the default for so long?
- What are the opportunities for innovation and new functionality in the metadata service?
The documentation suggests that Gravitino has overlap with a number of tool categories such as table schema (Hive metastore), metadata repository (Open Metadata), data federation (Trino/Alluxio). What are the capabilities that it can completely replace, and which will require other systems for more comprehensive functionality?
What are the capabilities that you are explicitly keeping out of scope for Gravitino?
Can you describe the technical architecture of Gravitino?
- How have the design and scope evolved from when you first started working on it?
Can you describe how Gravitino integrates into an overall data platform?
- In a typical day, what are the different ways that a data engineer or data analyst might interact with Gravitino?
One of the features that you highlight is centralized permissions management. Can you describe the access control model that you use for unifying across underlying sources?
What are the most interesting, innovative, or unexpected ways that you have seen Gravitino used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Gravitino?
When is Gravitino the wrong choice?
What do you have planned for the future of Gravitino?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.

Data Engineering Podcast

Enhancing Data Accessibility and Governance with Gravitino

Episode guests

Podcast summary created with Snipd AI

Quick takeaways

Deep dives

Unifying Data Catalogs

Integration of Structured and Unstructured Data

Addressing Legacy Systems and Compatibility

Enhancing Data Governance and Access Management

Remember Everything You Learn from Podcasts