O11ycast Ep. #79, AI and Otel: Look at your Data with Hamel Husain
Mar 18, 2025
Hamel Husain, a machine learning expert with over 25 years of experience at Airbnb and GitHub, dives deep into the world of AI and observability. He discusses the misleading nature of off-the-shelf metrics and highlights the critical role of error analysis in making AI models reliable. Listeners learn about the unique challenges in monitoring AI systems and the importance of insightful data analysis. Hamel also shares fascinating insights into Honeycomb's Query Assistant and how AI can evolve in coding practices.
AI Snips
Chapters
Transcript
Episode notes
Make Data Analysis Enjoyable
- Make the process of looking at data low-friction and enjoyable.
- Design tools that allow for easy navigation, note-taking, and rendering of various data formats.
Error Analysis for AI Observability
- Start by looking at your AI data systematically, even if it seems tedious.
- Write down any issues you find, focusing on what goes wrong, to gain intuition.
Honeycomb's Query Assistant
- Honeycomb built Query Assistant to help new users write queries using natural language.
- They prioritized creating runnable queries and iterated based on error analysis.