Paolo and Tim discuss the relevance of open AI and its impact on Europe, delve into the concept of data quality and highlight tools like pandera for addressing data quality issues. They also compare data quality tools and recommend SODA as their preferred tool. Additionally, they discuss the importance of data quality for AI and the challenges of building a long-lasting programming language. The hosts engage in a guessing game and debate Elon Musk's tweet about the zombie apocalypse.
Data quality tools like Soda and Great Expectations can ensure the accuracy and reliability of AI models.
Collaboration between developers and domain experts in defining data quality rules and expectations is crucial.
The choice of data quality tools depends on factors such as company size and requirements, with options like Soda, Great Expectations, and dbt tests offering different benefits.
Deep dives
Data Quality for AI Models
Data quality is not only relevant for datasets but also for AI models. Organizations need to ensure that AI models produce accurate and reliable results. Tools like Soda and Great Expectations can help check the accuracy and consistency of AI models. Additionally, the concept of data contracts is emerging, where stakeholders define agreements on how data sets and AI models should behave.
Integration with Development Workflow
Data quality tools, such as Great Expectations and dbt tests, can be integrated into the development workflow of data engineers. By implementing checks and tests during development, developers can ensure the validity and accuracy of data. It is essential for developers and domain experts to collaborate in defining data quality rules and expectations.
Choosing the Right Data Quality Tool
The choice of data quality tool depends on various factors. For tech startups building mobile applications, dbt tests and tools like Soda can be beneficial. These tools offer ease of use, integration with existing data warehouses, and reporting capabilities. For larger corporations, a comprehensive data quality framework that includes tools like Soda, Great Expectations, and dbt tests can provide governance, reporting, and scalability.
Data Contracts and Schema Validation
Data contracts, similar to API contracts, are becoming relevant for ensuring data quality. These contracts define the structure, expectations, and responsibilities of data producers and consumers. Tools like dbt tests and dbt support data contracts, enabling versioning and specification of the schema. Schema validation, integrated into the development process, ensures that data produced and consumed adheres to the defined contracts and expectations.
Snoop Dogg's Future Plans
When I'm no longer wrapping I want to open up an ice cream parlor and call myself scoop dog.
Elon Musk's Zombie Apocalypse Quotes
Forming an alliance with A.I. powered robot overlords to protect humanity against the undead uprising.
Welcome to another engaging episode of Datatopics Unplugged, the podcast where tech and relaxation intersect. Today, we're excited to host two special guests, Paolo and Tim, who bring their unique perspectives to our cozy corner.
Guests of Today
Paolo: An enthusiast of fantasy and sci-fi reading, Paolo is on a personal mission to reduce his coffee consumption. He has a unique way of measuring his height, at 0.89 Sams tall. With over two and a half years of experience as a data engineer at dataroots, Paolo contributes a rich professional perspective. His hobbies extend to playing field hockey and a preference for the warmer summer season.
Tim: Occasionally known as Dr. Dunkenstein, Tim brings a mix of humor and insight. He measures his height at 0.87 Sams tall. As the Head of Bizdev, he prefers to steer clear of grand titles, revealing his views on hierarchical structures and monarchies.
Data Quality Insights: A blog post by Paolo on data quality vs. data validation. We'll explore when and why data quality is essential, and evaluate tools like dbt, soda, deequ, and great_expectations: https://dataroots.io/blog/state-of-data-quality-october-2023
Join us for this mix of expert insights and light-hearted moments. Whether you're deeply embedded in the tech world or just dipping your toes in, this episode promises to be both informative and entertaining!
And, yes. There is a voucher, go to dataroots.io and navigate to the shop (top right) and use voucher code murilos_bargain_blast for a 25EUR discount!
Get the Snipd podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share & Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode