The Open Source Test Framework for Your Data Pipelines

Summary

Data pipelines are complicated and business critical pieces of technical infrastructure. Unfortunately they are also complex and difficult to test, leading to a significant amount of technical debt which contributes to slower iteration cycles. In this episode James Campbell describes how he helped create the Great Expectations framework to help you gain control and confidence in your data delivery workflows, the challenges of validating and monitoring the quality and accuracy of your data, and how you can use it in your own environments to improve your ability to move fast.

Announcements

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
• You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Corinium Global Intelligence, ODSC, and Data Council. Upcoming events include the Software Architecture Conference in NYC, Strata Data in San Jose, and PyCon US in Pittsburgh. Go to dataengineeringpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today.
• Your host is Tobias Macey and today I’m interviewing James Campbell about Great Expectations, the open source test framework for your data pipelines which helps you continually monitor and validate the integrity and quality of your data

Interview

• Introduction
• How did you get involved in the area of data management?
• Can you start by explaining what Great Expecations is and the origin of the project?
  • What has changed in the implementation and focus of Great Expectations since we last spoke on Podcast.__init__ 2 years ago?
• Prior to your introduction of Great Expectations what was the state of the industry with regards to testing, monitoring, or validation of the health and quality of data and the platforms operating on them?
• What are some of the types of checks and assertions that can be made about a pipeline using Great Expectations?
  • What are some of the non-obvious use cases for Great Expectations?
• What aspects of a data pipeline or the context that it operates in are unable to be tested or validated in a programmatic fashion?
• Can you describe how Great Expectations is implemented?
• For anyone interested in using Great Expectations, what is the workflow for incorporating it into their environments?
• What are some of the test cases that are often overlooked which data engineers and pipeline operators should be considering?
• Can you talk through some of the ways that Great Expectations can be extended?
• What are some notable extensions or integrations of Great Expectations?
• Beyond the testing and validation of data as it is being processed you have also included features that support documentation and collaboration of the data lifecycles. What are some of the ways that those features can benefit a team working with Great Expectations?
• What are some of the most interesting/innovative/unexpected ways that you have seen Great Expectations used?
• What are the limitations of Great Expectations?
• What are some cases where Great Expectations would be the wrong choice?
• What do you have planned for the future of Great Expectations?

Contact Info

• LinkedIn
• @jpcampbell42 on Twitter
• jcampbell on GitHub

Parting Question

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

• Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
• To help other people find the show please leave a review on iTunes and tell your friends and co-workers
• Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat

Links

• Great Expectations
  • GitHub
  • Twitter
• Podcast.__init__ Interview on Great Expectations
• Superconductive Health
• Abe Gong
• Pandas
  • Podcast.__init__ Interview
• SQLAlchemy
• PostgreSQL
  • Podcast Episode
• RedShift
• BigQuery
• Spark
• Cloudera
• DataBricks
• Great Expectations Data Docs
• Great Expectations Data Profiling
• Apache NiFi
• Amazon Deequ
• Tensorflow Data Validation

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA