Vanishing Gradients Episode 45: Your AI application is broken. Here’s what to do about it.
Feb 20, 2025
Joining the discussion is Hamel Husain, a seasoned ML engineer and open-source contributor, who shares invaluable insights on debugging generative AI systems. He emphasizes that understanding data is key to fixing broken AI applications. Hamel advocates for spreadsheet error analysis over complex dashboards. He also highlights the pitfalls of trusting LLM judges blindly and critiques existing AI dashboard metrics. His practical methods will transform how developers approach model performance and iteration in AI.
AI Snips
Chapters
Transcript
Episode notes
Look at Your Data
- Systematically look at your data to debug AI applications.
- Many people claim to look at their data, but don't do so effectively.
Spreadsheet Error Analysis
- Spreadsheet-based error analysis quickly uncovers failure modes.
- This low-tech approach clarifies what to work on next and how to measure it.
Start with Notes
- When starting error analysis, begin with simple notes.
- Avoid premature categorization; observe and document first.