MLOps Week 23: Data Quality & The Future of DataOps with Maxim Lukichev
Nov 14, 2023
auto_awesome
Maxim Lukichev, Co-founder and CTO at Telmai, discusses the importance of proactive data quality, improving collaboration on data teams, treating data as a product, data governance, and the impact of data on analytics, ML, and AI.
Data quality is crucial in preventing the corruption of outcomes and ensuring accurate results in data solutions, requiring collaboration among various teams.
Data observability is important due to the constant stream of data and the need to ensure the reliability and quality of the delivered data, necessitating effective teamwork and communication.
Deep dives
The Importance of Data Quality
Data quality is crucial in preventing the corruption of outcomes and ensuring accurate results in data solutions. Bad data can ruin processes and lead to costly resolutions. Telmine was inspired by the need to prevent bad data from entering systems, rather than fixing it afterwards. The complexity and variety of data make it a challenging problem to solve. Data quality issues span across multiple areas of an organization, requiring collaboration among various teams. Addressing data quality is both a technical and people problem, necessitating effective teamwork and communication.
The Significance of Data Observability
Data observability is important due to the constant stream of data and the need to ensure the reliability and quality of the delivered data. The complexity and velocity of data present challenges in identifying what constitutes good data. Traditional manual checks and rules are insufficient in a data-driven world. The problem spans from pipeline builders to data owners, and encompasses technical and organizational constraints. Implementing data observability tools can help automate routine actions, allowing teams to focus on more complex issues and root cause analysis.
Integration of Data Governance and Data Product Management
Organizations are moving towards treating data as a product, giving rise to roles such as data product owners and data product managers. Data as a product requires careful governance, access control, and provenance. The collaboration of data engineering, data product management, and other roles is crucial in building better data products. Consolidation and alignment among these roles will be beneficial, as too much fragmentation can hinder efficient and effective data operations. Data observability, data quality, and automation tools play a significant role in facilitating collaboration and enabling self-service data products within organizations.
For Episode 23 of the MLOps Weekly Podcast, Simba chats with Maxim Lukichev, Co-founder and CTO at Telmai. They discuss the importance of a proactive approach to data quality, improving collaboration on data teams, and the critical value of Data Ops in Large Language Models.
Get the Snipd podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share & Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode