Too many teams are building AI applications without truly understanding why their models fail. Instead of jumping straight to LLM evaluations, dashboards, or vibe checks, how do you actually fix a broken AI app?

In this episode, Hugo speaks with Hamel Husain, longtime ML engineer, open-source contributor, and consultant, about why debugging generative AI systems starts with looking at your data.

In this episode, we dive into:

• Why “look at your data” is the best debugging advice no one follows.
  
• How spreadsheet-based error analysis can uncover failure modes faster than complex dashboards.
  
• The role of synthetic data in bootstrapping evaluation.
  
• When to trust LLM judges—and when they’re misleading.
  
• Why most AI dashboards measuring truthfulness, helpfulness, and conciseness are often a waste of time.
  

If you're building AI-powered applications, this episode will change how you approach debugging, iteration, and improving model performance in production.

LINKS

• The podcast livestream on YouTube
• Hamel's blog
• Hamel on twitter
• Hugo on twitter
• Vanishing Gradients on twitter
• Vanishing Gradients on YouTube
• Vanishing Gradients on Twitter

• Vanishing Gradients on Lu.ma

• Building LLM Application for Data Scientists and SWEs, Hugo course on Maven (use VG25 code for 25% off)

• Hugo is also running a free lightning lesson next week on LLM Agents: When to Use Them (and When Not To)