Eval strategy and human-in-the-loop scoring

Guests:

• Thom van der Doef, Principal Product Designer at eSpark
• Mary [last name], Director of Learning Design & Product Manager at eSpark
• Ray Lyons, VP of Product & Engineering at eSpark

Topics covered:

• The origin story of Teacher Assistant: connecting administrator mandates with teacher needs
• Why the team abandoned a chatbot interface in favor of a more structured workflow
• How retrieval augmented generation (RAG) and embeddings shaped the product architecture
• Lessons learned from debugging semantic search vs. keyword search
• Building evals with rubrics, Braintrust, and a human-in-the-loop approach
• What’s next for Teacher Assistant: more contextual recommendations using student data

Links & References:

• eSpark Learning
• Braintrust – evals and observability for LLM applications
• AI Evals Course by Hamel Husain and Shreya Shanker (Get 35% off with my affiliate link)

Chapters: 02:05 Overview of Epar's Adaptive Learning Program 07:19 Challenges and Insights from COVID-19 17:06 Developing the Teacher Assistant Feature 24:55 User Experience and Interface Evolution 34:29 Chat GPT-5's New Features 35:16 Balancing Engagement and Efficiency 35:40 Seasonal Business and Real Traffic 36:29 Technical Decisions and RAG Implementation 38:28 Challenges with Embeddings and Metadata 41:24 Improving Recommendations and Data Enrichment 55:18 Evaluating the Teaching Assistant 01:05:51 Future Plans and User Feedback 01:07:57 Conclusion and Final Thoughts