Eliminate Friction In Your Data Platform Through Unified Metadata Using OpenMetadata
Data Engineering Podcast
Discovering All Your Data Assets in a Single Is a Problem
The open medadata project is trying to be this kind of universal approach to meda data. It aims to integrate all the different components of the data ecosystem, which is an intractable problem because every system has some type of database. And so being able to expose that and unlock it and share it was a very complex and multifacited problem. So we started building lot of automation, lata tooling, improved consuming the meta data. A lot of tools became lots simpler to build because there a centralized metadata store. We decided that we will come and build it outside as an open source because we have a lot of open source background.
Summary
A significant source of friction and wasted effort in building and integrating data management systems is the fragmentation of metadata across various tools. After experiencing the impacts of fragmented metadata and previous attempts at building a solution Suresh Srinivas and Sriharsha Chintalapani created the OpenMetadata project. In this episode they share the lessons that they have learned through their previous attempts and the positive impact that a unified metadata layer had during their time at Uber. They also explain how the OpenMetadat project is aiming to be a common standard for defining and storing metadata for every use case in data platforms and the ways that they are architecting the reference implementation to simplify its adoption. This is an ambitious and exciting project, so listen and try it out today.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
- Struggling with broken pipelines? Stale dashboards? Missing data? If this resonates with you, you’re not alone. Data engineers struggling with unreliable data need look no further than Monte Carlo, the world’s first end-to-end, fully automated Data Observability Platform! In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Monte Carlo monitors and alerts for data issues across your data warehouses, data lakes, ETL, and business intelligence, reducing time to detection and resolution from weeks or days to just minutes. Start trusting your data with Monte Carlo today! Visit dataengineeringpodcast.com/impact today to save your spot at IMPACT: The Data Observability Summit a half-day virtual event featuring the first U.S. Chief Data Scientist, founder of the Data Mesh, Creator of Apache Airflow, and more data pioneers spearheading some of the biggest movements in data. The first 50 to RSVP with this link will be entered to win an Oculus Quest 2 — Advanced All-In-One Virtual Reality Headset. RSVP today – you don’t want to miss it!
- Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Hightouch is the easiest way to sync data into the platforms that your business teams rely on. The data you’re looking for is already in your data warehouse and BI tools. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems. No more scripts, just SQL. Supercharge your business teams with customer data using Hightouch for Reverse ETL today. Get started for free at dataengineeringpodcast.com/hightouch.
- Your host is Tobias Macey and today I’m interviewing Sriharsha Chintalapani and Suresh Srinivas about OpenMetadata, an open standard for metadata and a reference implementation for a central metadata store
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what the OpenMetadata project is and the story behind it?
- What are the goals of the project?
- What are the common challenges faced by engineers and data practitioners in organizing the metadata for their systems?
- What are the capabilities that a centralized and holistic view of a platform’s metadata can enable?
- How would you characterize the current state and progress on the open source initiative around OpenMetadata?
- How does OpenMetadata compare to the OpenLineage project and other similar systems?
- What opportunities do you see for collaborating with or learning from their efforts?
- What are the schema elements that you have identified as critical to a holistic view of an organization’s metadata?
- For an organization with an existing data platform, what is the role that OpenMetadata plays, and what are the points of integration across the different components?
- Can you describe the implementation of the OpenMetadata architecture?
- What are the user experience and operational characteristics that you are trying to optimize for as you iterate on the project?
- What are the challenges that you face in balancing the generality and specificity of the core schemas for metadata objects?
- There are a large and growing number of businesses that create systems on top of an organizations metadata in the form of catalogs, observability, governance, data quality, etc. What do you see as the role of the OpenMetadata project across that ecosystem of products?
- How has your perspective on the domain of metadata management and the associated challenges changed or evolved as you have been working on this project?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on OpenMetadata?
- When is OpenMetadata the wrong choice?
- What do you have planned for the future of OpenMetadata?
Contact Info
- Suresh
- @suresh_m_s on Twitter
- sureshms on GitHub
- Sriharsha
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
- OpenMetadata
- Apache Storm
- Apache Kafka
- Hortonworks
- Apache Atlas
- OpenMetadata Sandbox
- OpenLineage
- Egeria
- JSON Schema
- Amundsen
- DataHub
- JanusGraph
- Titan Graph Database
- HBase
- Jetty
- DropWizard
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Remember Everything You Learn from Podcasts
