Building A System Of Record For Your Organization's Data Ecosystem At Metaphor
Data Engineering Podcast
00:00
The Universal Substrait for Meda Data
The open meditator project is trying to be this universal sub strait for medadata in a data ecosystem. The idea is you want to be strongly schematized, but not to the extent where you essentially become the least common denominator. So how do you have a stronge schemead hig system and also same time very extendable, isnt? In anof iself? Its a very interesting engineering challenge.
Transcript
Play full episode
Transcript
Episode notes
Summary
Building a well managed data ecosystem for your organization requires a holistic view of all of the producers, consumers, and processors of information. The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data. In this episode Pardhu Gunnam and Mars Lan explain how they have designed the architecture and user experience to allow everyone to collaborate on the data lifecycle and provide opportunities for automation and extensible workflows.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
- Struggling with broken pipelines? Stale dashboards? Missing data? If this resonates with you, you’re not alone. Data engineers struggling with unreliable data need look no further than Monte Carlo, the world’s first end-to-end, fully automated Data Observability Platform! In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Monte Carlo monitors and alerts for data issues across your data warehouses, data lakes, ETL, and business intelligence, reducing time to detection and resolution from weeks or days to just minutes. Start trusting your data with Monte Carlo today! Visit dataengineeringpodcast.com/montecarlo to learn more. The first 10 people to request a personalized product tour will receive an exclusive Monte Carlo Swag box.
- Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Hightouch is the easiest way to sync data into the platforms that your business teams rely on. The data you’re looking for is already in your data warehouse and BI tools. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems. No more scripts, just SQL. Supercharge your business teams with customer data using Hightouch for Reverse ETL today. Get started for free at dataengineeringpodcast.com/hightouch.
- Your host is Tobias Macey and today I’m interviewing Pardhu Gunnam and Mars Lan about Metaphor Data, a platform aiming to be the system of record for your data ecosystem
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Metaphor is and the story behind it?
- On your site it states that you are aiming to be the "system of record" for your data platform. Can you unpack that statement and its implications?
- What are the shortcomings in the "data catalog" approach to metadata collection and presentation?
- Who are the target end users of Metaphor and what are the pain points for each persona that you are prioritizing?
- How has that focus informed your priorities for user experience design and feature development?
- Can you describe how the Metaphor platform is architected?
- What are the lessons that you learned from your work at DataHub that have informed your work on Metaphor?
- There has been a huge amount of focus on the "modern data stack" with an assumption that there is a cloud data warehouse as the central component that all data flows through. How does Metaphor’s design allow for usage in platforms that aren’t dominated by a cloud data warehouse?
- What are some examples of information that you can extract through integrations with an organization’s communication platforms?
- Can you talk through a few example workflows where that information is used to inform the actions taken by a team member?
- What is your philosophy around data modeling or schema standardization for metadata records?
- What are some of the challenges that teams face in stitching together a meaningful set of relations across metadata records in Metaphor?
- What are some of the features or potential use cases for Metaphor that are overlooked or misunderstood as you work with your customers?
- What are the most interesting, innovative, or unexpected ways that you have seen Metaphor used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Metaphor?
- When is Metaphor the wrong choice?
- What do you have planned for the future of Metaphor?
Contact Info
- Pardhu
- @PardhuGunnam on Twitter
- Mars
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
- Metaphor
- DataHub
- Transform
- Supergrain
- MetriQL
- dbt
- OpenMetadata
- Pegasus Data Language
- Modern Data Experience
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
The AI-powered Podcast Player
Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!
