Just Now Possible Debugging AI Products: From Data Leakage to Evals with Hamel Husain
17 snips
Oct 2, 2025 
Hamel Husain, a machine learning engineer with over 25 years of experience at GitHub and Airbnb, dives deep into the intricacies of debugging AI products. He shares insights from his work on forecasting Airbnb guest growth, highlighting challenges like data leakage. The conversation uncovers techniques for error analysis in machine learning, the importance of synthetic data, and the pitfalls of AI-generated outputs like hallucinations. Hamel emphasizes the need for systematic improvement and presents practical tips for enhancing AI evaluations.
AI Snips
Chapters
Transcript
Episode notes
Debugging Is The Core Of ML Work
- ML product work is mostly about debugging data and measurement, not just models.
- Hamel Husain stresses scientific skepticism and iterative measurement to improve AI systems.
ML Is Prediction, Not Magic
- Machine learning is fundamentally predictive: classification, recommendation, forecasting.
- The core challenge is making models generalize to unseen data, not complexity of algorithms.
Airbnb Forecasts Revealed Data Leakage
- At Airbnb Hamel built guest lifetime value forecasts and found overly-accurate results.
- He discovered data leakage from future signals and had to debug messy production data.