Build Better Tests For Your dbt Projects With Datafold And data-diff
Data Engineering Podcast
The Gap in the Tooling and Technology That's Available for Data Management
I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today. I think in general, I believe that people should be able to do more with data. Data engineers are doing way too many things manually. They're writing too much code. And it's not necessarily a single gap. It's probably a thousand different small gaps that exist that we really aim to solve for data practitioners and data developers.
Summary
Data engineering is all about building workflows, pipelines, systems, and interfaces to provide stable and reliable data. Your data can be stable and wrong, but then it isn't reliable. Confidence in your data is achieved through constant validation and testing. Datafold has invested a lot of time into integrating with the workflow of dbt projects to add early verification that the changes you are making are correct. In this episode Gleb Mezhanskiy shares some valuable advice and insights into how you can build reliable and well-tested data assets with dbt and data-diff.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Sign up free at dataengineeringpodcast.com/rudderstack
- Your host is Tobias Macey and today I'm interviewing Gleb Mezhanskiy about how to test your dbt projects with Datafold
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Datafold is and what's new since we last spoke? (July 2021 and July 2022 about data-diff)
- What are the roadblocks to data testing/validation that you see teams run into most often?
- How does the tooling used contribute to/help address those roadblocks?
- What are some of the error conditions/failure modes that data-diff can help identify in a dbt project?
- What are some examples of tests that need to be implemented by the engineer?
- In your experience working with data teams, what typically constitutes the "staging area" for a dbt project? (e.g. separate warehouse, namespaced tables, snowflake data copies, lakefs, etc.)
- Given a dbt project that is well tested and has data-diff as part of the validation suite, what are the challenges that teams face in managing the feedback cycle of running those tests?
- In application development there is the idea of the "testing pyramid", consisting of unit tests, integration tests, system tests, etc. What are the parallels to that in data projects?
- What are the limitations of the data ecosystem that make testing a bigger challenge than it might otherwise be?
- Beyond test execution, what are the other aspects of data health that need to be included in the development and deployment workflow of dbt projects? (e.g. freshness, time to delivery, etc.)
- What are the most interesting, innovative, or unexpected ways that you have seen Datafold and/or data-diff used for testing dbt projects?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on dbt testing internally or with your customers?
- When is Datafold/data-diff the wrong choice for dbt projects?
- What do you have planned for the future of Datafold?
Contact Info
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
- Datafold
- data-diff
- dbt
- Dagster
- dbt-cloud slim CI
- GitHub Actions
- Jenkins
- Circle CI
- Dolt
- Malloy
- LakeFS
- Planetscale
- Snowflake Zero Copy Cloning
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Special Guest: Gleb Mezhanskiy.
Sponsored By:
- Rudderstack:  RudderStack provides all your customer data pipelines in one platform. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. RudderStack’s warehouse-first approach means it does not store sensitive information, and it allows you to leverage your existing data warehouse/data lake infrastructure to build a single source of truth for every team. RudderStack also supports real-time use cases. You can Implement RudderStack SDKs once, then automatically send events to your warehouse and 150+ business tools, and you’ll never have to worry about API changes again. Visit [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack) to sign up for free today, and snag a free T-Shirt just for being a Data Engineering Podcast listener.
Remember Everything You Learn from Podcasts
