O11ycast Ep. #79, AI and Otel: Look at your Data with Hamel Husain
Mar 18, 2025
Hamel Husain, a machine learning expert with over 25 years of experience at Airbnb and GitHub, dives deep into the world of AI and observability. He discusses the misleading nature of off-the-shelf metrics and highlights the critical role of error analysis in making AI models reliable. Listeners learn about the unique challenges in monitoring AI systems and the importance of insightful data analysis. Hamel also shares fascinating insights into Honeycomb's Query Assistant and how AI can evolve in coding practices.
AI Snips
Chapters
Transcript
Episode notes
Error Analysis for AI Observability
- Start by looking at your AI data systematically, even if it seems tedious.
- Write down any issues you find, focusing on what goes wrong, to gain intuition.
Make Data Analysis Enjoyable
- Make the process of looking at data low-friction and enjoyable.
- Design tools that allow for easy navigation, note-taking, and rendering of various data formats.
Prioritizing Runnable Queries
- Prioritize runnable queries over perfect ones, especially for new users.
- Focus on engaging users and enabling iteration over one-shot perfection.