Data Engineering Podcast Build A Common Understanding Of Your Data Reliability Rules With Soda Core and Soda Checks Language
41:02
Tax Reporting Example
- Imagine a bank needing to report customer tax information due to legal requirements.
- A mobile team update accidentally removes a crucial tax field, risking legal action.
Data Quality's Importance
- Data quality is crucial for businesses, as data issues can severely impact operations and legal compliance.
- Data connects different teams, and quality ensures smooth data flow and reliable insights.
Data Quality Action Plan
- To address data quality, establish clear systems and processes.
- Implement three steps: identify issues, perform root cause analysis, and resolve issues.
Get the Snipd Podcast app to discover more snips from this episode
Get the app 1 chevron_right 2 chevron_right 3 chevron_right 4 chevron_right 5 chevron_right 6 chevron_right 7 chevron_right 8 chevron_right 9 chevron_right 10 chevron_right 11 chevron_right 12 chevron_right 13 chevron_right 14 chevron_right 15 chevron_right
Introduction
00:00 • 3min
The Importance of Data Quality for the Business
02:41 • 5min
Resolving the Data Issues
07:25 • 1min
Sora Cl Data Quality
08:53 • 4min
Scaling Data Quality Towards the Enterprise?
13:20 • 3min
Hebo Data - Data Pipeline Platform
16:02 • 5min
Ora Cloud - What's the Investment in Syntax Highlighting and Editor Help?
20:44 • 3min
Soda Cor Library
23:22 • 1min
SodaCore
24:51 • 2min
SodaCors
26:35 • 2min
Capsilating Reusable Constructs in a Programming Language
28:43 • 4min
Data Engineering Podcast - Slash Prefect to Day
32:22 • 2min
How Do You See the Next Big Step?
34:47 • 2min
The Biggest Lesson Learned From Monitoring Data Quality
36:55 • 2min
The Biggest Gap in the Data Technology Stack
38:34 • 2min
Summary
Regardless of how data is being used, it is critical that the information is trusted. The practice of data reliability engineering has gained momentum recently to address that question. To help support the efforts of data teams the folks at Soda Data created the Soda Checks Language and the corresponding Soda Core utility that acts on this new DSL. In this episode Tom Baeyens explains their reasons for creating a new syntax for expressing and validating checks for data assets and processes, as well as how to incorporate it into your own projects.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
- Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value. Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos.
- Prefect is the modern Dataflow Automation platform for the modern data stack, empowering data practitioners to build, run and monitor robust pipelines at scale. Guided by the principle that the orchestrator shouldn’t get in your way, Prefect is the only tool of its kind to offer the flexibility to write code as workflows. Prefect specializes in glueing together the disparate pieces of a pipeline, and integrating with modern distributed compute libraries to bring power where you need it, when you need it. Trusted by thousands of organizations and supported by over 20,000 community members, Prefect powers over 100MM business critical tasks a month. For more information on Prefect, visit dataengineeringpodcast.com/prefect.
- Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day. Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc., are already available as plug-and-play connectors with reliable, intuitive SaaS solutions. Hevo Data is a highly reliable and intuitive data pipeline platform used by data engineers from 40+ countries to set up and run low-latency ELT pipelines with zero maintenance. Boasting more than 150 out-of-the-box connectors that can be set up in minutes, Hevo also allows you to monitor and control your pipelines. You get: real-time data flow visibility, fail-safe mechanisms, and alerts if anything breaks; preload transformations and auto-schema mapping precisely control how data lands in your destination; models and workflows to transform data for analytics; and reverse-ETL capability to move the transformed data back to your business software to inspire timely action. All of this, plus its transparent pricing and 24*7 live support, makes it consistently voted by users as the Leader in the Data Pipeline category on review platforms like G2. Go to dataengineeringpodcast.com/hevodata and sign up for a free 14-day trial that also comes with 24×7 support.
- Your host is Tobias Macey and today I’m interviewing Tom Baeyens about Soda Data’s new DSL for data reliability
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what SodaCL is and the story behind it?
- What is the scope of functionality that SodaCL is intended to address?
- What are the ways that reliability is measured for data assets? (what is the equivalent to site uptime?)
- What are the core abstractions that you identified for simplifying the declaration of data validations?
- How did you approach the design of the SodaCL syntax to balance flexibility for various use cases, with structure and opinionated application?
- Why YAML?
- Can you describe how the Soda Core utility is implemented?
- How have the design and scope of the SodaCL dialect and the Soda Core framework evolved since you started working on them?
- What are the available integration/extension points for teams who are using Soda Core?
- Can you describe how SodaCL integrates into the workflow of data and analytics engineers?
- What is your process for evolving the SodaCL dialect in a maintainable and sustainable manner?
- What are the most interesting, innovative, or unexpected ways that you have seen SodaCL used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on SodaCL?
- When is SodaCL the wrong choice?
- What do you have planned for the future of SodaCL?
Contact Info
- @tombaeyens on Twitter
- tombaeyens on GitHub
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don’t forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
Links
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA