Data Engineering Podcast Collecting And Retaining Contextual Metadata For Powerful And Effective Data Discovery
53:24
Data Discovery vs. Data Catalogs
- Data discovery is about finding and understanding data, not just cataloging it.
- It requires context like usage patterns, origin, and relationships, which data catalogs often lack.
Effective Data Discovery
- Consolidate metadata and provide context like data origin and usage.
- Encourage social interaction and knowledge sharing around data assets.
Social Data Discovery
- Integrate data discovery with communication channels like Slack or email.
- Capture tribal knowledge and discussions about data for searchability.
Get the Snipd Podcast app to discover more snips from this episode
Get the app 1 chevron_right 2 chevron_right 3 chevron_right 4 chevron_right 5 chevron_right 6 chevron_right 7 chevron_right 8 chevron_right 9 chevron_right 10 chevron_right 11 chevron_right 12 chevron_right 13 chevron_right 14 chevron_right 15 chevron_right 16 chevron_right 17 chevron_right 18 chevron_right 19 chevron_right 20 chevron_right
Introduction
00:00 • 3min
Data Discovery
02:51 • 2min
The Social Component of Data Discovery
05:03 • 3min
Data Catalogue Versus Data Discovery
07:57 • 3min
The Evolution of Medadata Catalogues
10:59 • 3min
How Do I Find What I'm Looking For?
13:45 • 2min
How to Discover Other Data Sets
16:12 • 3min
Select Star - Data Discovery Platform
18:58 • 3min
Using Foreign Key Relationships in a Data Warehouse Environment
21:37 • 3min
Getting the Right Relationships From Source to Destination
24:16 • 3min
Data Engineering Podcast - Ascend Data Automation Cloud
26:57 • 2min
Using a Data Lineage Model in a Bi Looker
29:24 • 3min
Select Star Data Catalogue - What's Next?
32:07 • 3min
Select Star - A Data Discovery Platform for Data Warehouses
35:03 • 4min
Using Data Discovery Capabilities in a Data Discovery Environment
38:49 • 2min
Using Lineagea Pi to Generate Reports
40:35 • 3min
What Are Some of the Most Interesting or Unexpected Lessons That You've Learned?
43:39 • 2min
The Importance of Data Discovery
45:56 • 3min
Is There a Biggest Gap in Data Management?
49:00 • 3min
Data Engineering Podcast - Thank You Sabius
51:34 • 2min
Summary
Data is useless if it isn’t being used, and you can’t use it if you don’t know where it is. Data catalogs were the first solution to this problem, but they are only helpful if you know what you are looking for. In this episode Shinji Kim discusses the challenges of data discovery and how to collect and preserve additional context about each piece of information so that you can find what you need when you don’t even know what you’re looking for yet.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
- Data stacks are becoming more and more complex. This brings infinite possibilities for data pipelines to break and a host of other issues, severely deteriorating the quality of the data and causing teams to lose trust. Sifflet solves this problem by acting as an overseeing layer to the data stack – observing data and ensuring it’s reliable from ingestion all the way to consumption. Whether the data is in transit or at rest, Sifflet can detect data quality anomalies, assess business impact, identify the root cause, and alert data teams’ on their preferred channels. All thanks to 50+ quality checks, extensive column-level lineage, and 20+ connectors across the Data Stack. In addition, data discovery is made easy through Sifflet’s information-rich data catalog with a powerful search engine and real-time health statuses. Listeners of the podcast will get $2000 to use as platform credits when signing up to use Sifflet. Sifflet also offers a 2-week free trial. Find out more at dataengineeringpodcast.com/sifflet today!
- The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it. Select Star’s data discovery platform solves that out of the box, with an automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day. Just connect it to your database/data warehouse/data lakehouse/whatever you’re using and let them do the rest. Go to dataengineeringpodcast.com/selectstar today to double the length of your free trial and get a swag package when you convert to a paid plan.
- Data teams are increasingly under pressure to deliver. According to a recent survey by Ascend.io, 95% in fact reported being at or over capacity. With 72% of data experts reporting demands on their team going up faster than they can hire, it’s no surprise they are increasingly turning to automation. In fact, while only 3.5% report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. 85%!!! That’s where our friends at Ascend.io come in. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Go to dataengineeringpodcast.com/ascend and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.
- Your host is Tobias Macey and today I’m interviewing Shinji Kim about data discovery and what is required to build and maintain useful context for your information assets
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you share your definition of "data discovery" and the technical/social/process components that are required to make it viable?
- What are the differences between "data discovery" and the capabilities of a "data catalog" and how do they overlap?
- discovery of assets outside the bounds of the warehouse
- capturing and codifying tribal knowledge
- creating a useful structure/framework for capturing data context and operationalizing it
- What are the most interesting, innovative, or unexpected ways that you have seen data discovery implemented?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on data discovery at SelectStar?
- When might a data discovery effort be more work than is required?
- What do you have planned for the future of SelectStar?
Contact Info
- @shinjikim on Twitter
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don’t forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
Links
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA