Unsupervised Learning Ep 28: LangChain CEO Harrison Chase on the Current State of Eval and Agents and The LLM Apps that Will Define 2024
88 snips
Feb 20, 2024 
LangChain CEO Harrison Chase discusses AI evaluation, agent landscape, open vs. closed source models, and future of AI applications. Topics include LangSmith, AI in sports, evaluation practices, and the potential of open-source models. The podcast also explores the current AI landscape, deploying LangChain applications with LangServe, and the evolution of more complex chatbots.
AI Snips
Chapters
Transcript
Episode notes
AI Sports Commentary Potential
- Creative applications like AI-generated sports commentary can personalize and scale engagement.
- This approach could revolutionize viewer experience without needing live commentators physically present.
Evaluation Requires Human-in-Loop
- Building an evaluation data set is crucial as it forces you to define what your system should do clearly.
- Human-in-the-loop evaluation remains important despite the dream of fully automated LLM self-evaluation.
Build and Use Eval Data Sets
- Look closely at real examples to understand system performance better.
- Spend time developing a representative evaluation data set before scaling your AI product.