AI Stories Post Deployment Data Science with Wojtek Kuberski #56
Feb 13, 2025
Wojtek Kuberski, Co-Founder and CTO at NannyML, shares his expertise in AI and data science. He discusses the challenges of model monitoring post-deployment, including covariate shift and concept drift. Wojtek explains how NannyML's algorithms assess model performance without needing labels. He emphasizes the critical role of continuous monitoring to prevent silent failures that could impact businesses. The conversation also touches on his transition from freelancing to building a product-focused company in the ever-evolving data science landscape.
AI Snips
Chapters
Transcript
Episode notes
Early Freelance Projects
- Wojtek Kuberski shared early freelance projects, including document classification and computer vision tasks like background removal.
- A gaming laptop bypassed IT requirements for a computer vision project due to its strong GPU.
Model Failure and Monitoring
- Models fail when real-world data distributions shift from training data (covariate shift).
- Monitoring is crucial because models are not trained for all possible data variations.
Concept Drift and Label Delay
- Besides covariate shift, concept drift causes model failure when input-target relationships change.
- Monitoring performance is challenging without production labels, especially in scenarios like credit scoring with delayed feedback.