Data Engineering Podcast Building A System Of Record For Your Organization's Data Ecosystem At Metaphor
Dec 20, 2021
01:05:34
Metaphor Data's Origin
- Metaphor Data's origin lies in LinkedIn's metadata team's work on data discovery and GDPR compliance.
- The team realized the metadata platform's potential beyond LinkedIn, leading to Metaphor Data's creation.
System of Record vs. Data Catalog
- Metaphor Data aims to be a system of record, aggregating metadata for consistent views.
- It breaks down data silos, improves consistency, and version control, unlike traditional data catalogs.
Target Personas
- Metaphor Data targets three personas: data producers, data consumers, and data leadership.
- It addresses data democratization, trust, resource optimization, and ROI concerns across these groups.
Get the Snipd Podcast app to discover more snips from this episode
Get the app 1 chevron_right 2 chevron_right 3 chevron_right 4 chevron_right 5 chevron_right 6 chevron_right 7 chevron_right 8 chevron_right 9 chevron_right 10 chevron_right 11 chevron_right 12 chevron_right 13 chevron_right 14 chevron_right 15 chevron_right 16 chevron_right 17 chevron_right 18 chevron_right 19 chevron_right 20 chevron_right
Introduction
00:00 • 2min
Metaphor Data, a Platform Aiming to Be the System of Record for Your Data Ecosystem
01:31 • 3min
Data Engineering - What's the Story Behind That?
04:20 • 5min
Metadata Platform
09:30 • 3min
The Universal Substrait for Meda Data
12:48 • 4min
Are You Building a Metaphor Platform for a Data Productor?
16:36 • 5min
The Architecture of a Data Hub Platform?
21:20 • 3min
The Reliability of a Platform
24:16 • 4min
Howdyou Build a Team Now Ago?
28:32 • 3min
Data Engineering Podcast - Monte Carlo
31:16 • 5min
Using a Data Stack in a Machine Learning Environment?
36:17 • 4min
Using Slack as a Data Platform?
39:52 • 3min
How to Simplify the Integration Process?
42:40 • 3min
The Challenges of Integrating With Metadata
45:28 • 3min
Using Metadata for Data Discovery?
48:13 • 3min
What Are Some of the Surprising Aspects of the Usage of MEDDATA?
51:14 • 3min
How Do You Use Data Relevant Knowledge?
54:06 • 3min
Building Metaphor, What Are Some of the Most Unexpected Lessons Learned?
57:19 • 3min
Metaphor Platform - What's Next?
01:00:18 • 2min
The Biggest Gap in Data Management
01:02:47 • 3min
Summary
Building a well managed data ecosystem for your organization requires a holistic view of all of the producers, consumers, and processors of information. The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data. In this episode Pardhu Gunnam and Mars Lan explain how they have designed the architecture and user experience to allow everyone to collaborate on the data lifecycle and provide opportunities for automation and extensible workflows.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
- Struggling with broken pipelines? Stale dashboards? Missing data? If this resonates with you, you’re not alone. Data engineers struggling with unreliable data need look no further than Monte Carlo, the world’s first end-to-end, fully automated Data Observability Platform! In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Monte Carlo monitors and alerts for data issues across your data warehouses, data lakes, ETL, and business intelligence, reducing time to detection and resolution from weeks or days to just minutes. Start trusting your data with Monte Carlo today! Visit dataengineeringpodcast.com/montecarlo to learn more. The first 10 people to request a personalized product tour will receive an exclusive Monte Carlo Swag box.
- Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Hightouch is the easiest way to sync data into the platforms that your business teams rely on. The data you’re looking for is already in your data warehouse and BI tools. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems. No more scripts, just SQL. Supercharge your business teams with customer data using Hightouch for Reverse ETL today. Get started for free at dataengineeringpodcast.com/hightouch.
- Your host is Tobias Macey and today I’m interviewing Pardhu Gunnam and Mars Lan about Metaphor Data, a platform aiming to be the system of record for your data ecosystem
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Metaphor is and the story behind it?
- On your site it states that you are aiming to be the "system of record" for your data platform. Can you unpack that statement and its implications?
- What are the shortcomings in the "data catalog" approach to metadata collection and presentation?
- Who are the target end users of Metaphor and what are the pain points for each persona that you are prioritizing?
- How has that focus informed your priorities for user experience design and feature development?
- Can you describe how the Metaphor platform is architected?
- What are the lessons that you learned from your work at DataHub that have informed your work on Metaphor?
- There has been a huge amount of focus on the "modern data stack" with an assumption that there is a cloud data warehouse as the central component that all data flows through. How does Metaphor’s design allow for usage in platforms that aren’t dominated by a cloud data warehouse?
- What are some examples of information that you can extract through integrations with an organization’s communication platforms?
- Can you talk through a few example workflows where that information is used to inform the actions taken by a team member?
- What is your philosophy around data modeling or schema standardization for metadata records?
- What are some of the challenges that teams face in stitching together a meaningful set of relations across metadata records in Metaphor?
- What are some of the features or potential use cases for Metaphor that are overlooked or misunderstood as you work with your customers?
- What are the most interesting, innovative, or unexpected ways that you have seen Metaphor used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Metaphor?
- When is Metaphor the wrong choice?
- What do you have planned for the future of Metaphor?
Contact Info
- Pardhu
- @PardhuGunnam on Twitter
- Mars
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
- Metaphor
- DataHub
- Transform
- Supergrain
- MetriQL
- dbt
- OpenMetadata
- Pegasus Data Language
- Modern Data Experience
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA