When to write throwaway tests versus permanent evals

Guest: Hamel Husain

AI products and problems discussed:

• GitHub Copilot
• Forecasting AirBnB Guest Growth - NurtureBoss

Resources & Links

• Hamel’s blog on AI evals
• AI Evals for Engineers and PMs course on Maven (Get 35% off with this affiliate link)

Chapters: 00:00 Introduction to Hamel Hussein 00:34 Challenges in AI Consulting 02:00 Machine Learning Fundamentals 04:47 Debugging Machine Learning Models 05:00 Case Study: Airbnb's Guest Growth 08:51 Understanding Machine Learning Models 18:35 Introduction to Nurture Boss 25:40 Building AI Products with Synthetic Data 41:20 Connecting Machine Learning to Error Analysis 42:28 Real-World Example: Text Message Errors 44:15 Prioritizing and Documenting Errors 45:59 Continuous Improvement and Iteration 58:08 Using Synthetic Data for Evaluation 01:08:42 Avoiding Overfitting in Evaluations 01:19:28 Practical Tips for Error Analysis 01:25:10 Final Thoughts and Resources