In this insightful discussion, Jineet Doshi, an award-winning AI lead with over seven years at Intuit, dives deep into the complexities of evaluating generative AI systems. He emphasizes the importance of holistic evaluation to foster trust and the unique challenges posed by large language models. Jineet explores diverse evaluation methods, from classic NLP techniques to innovative strategies like red teaming. He also tackles the financial nuances of generative AI and the balance between human insight and automated feedback for robust assessments.
57:33
forum Ask episode
web_stories AI Snips
view_agenda Chapters
auto_awesome Transcript
info_circle Episode notes
insights INSIGHT
LLM Evaluation Challenges
Evaluating LLMs is challenging due to their open-ended outputs and broad capabilities.
Traditional ML metrics are inadequate for assessing nuanced tasks like poem generation.
insights INSIGHT
Traditional NLP Techniques for LLM Evaluation
Traditional NLP techniques can be applied to LLM evaluation by using multiple-choice questions or text similarity.
However, these methods have limitations in evaluating open-ended tasks and can be sensitive to the choice of embedding models.
volunteer_activism ADVICE
Using Benchmarks for LLM Evaluation
Use benchmarks to evaluate LLMs across various factors like knowledge, reasoning, and toxicity.
Be mindful of benchmark limitations, data leakage, and the need for custom benchmarks for specific use cases.
Get the Snipd Podcast app to discover more snips from this episode
Jineet Doshi is an award-winning Scientist, Machine Learning Engineer, and Leader at Intuit with over 7 years of experience. He has a proven track record of leading successful AI projects and building machine-learning models from design to production across various domains which have impacted 100 million customers and significantly improved business metrics, leading to millions of dollars of impact.
Holistic Evaluation of Generative AI Systems // MLOps Podcast #280 with Jineet Doshi, Staff AI Scientist or AI Lead at Intuit.
// Abstract
Evaluating LLMs is essential in establishing trust before deploying them to production. Even post deployment, evaluation is essential to ensure LLM outputs meet expectations, making it a foundational part of LLMOps. However, evaluating LLMs remains an open problem. Unlike traditional machine learning models, LLMs can perform a wide variety of tasks such as writing poems, Q&A, summarization etc. This leads to the question how do you evaluate a system with such broad intelligence capabilities? This talk covers the various approaches for evaluating LLMs such as classic NLP techniques, red teaming and newer ones like using LLMs as a judge, along with the pros and cons of each. The talk includes evaluation of complex GenAI systems like RAG and Agents. It also covers evaluating LLMs for safety and security and the need to have a holistic approach for evaluating these very capable models.
// Bio
Jineet Doshi is an award winning AI Lead and Engineer with over 7 years of experience. He has a proven track record of leading successful AI projects and building machine learning models from design to production across various domains, which have impacted millions of customers and have significantly improved business metrics, leading to millions of dollars of impact. He is currently an AI Lead at Intuit where he is one of the architects and developers of their Generative AI platform, which is serving Generative AI experiences for more than 100 million customers around the world.
Jineet is also a guest lecturer at Stanford University as part of their building LLM Applications class. He is on the Advisory Board of University of San Francisco’s AI Program. He holds multiple patents in the field, is on the steering committee of MLOps World Conference and has also co chaired workshops at top AI conferences like KDD. He holds a Masters degree from Carnegie Mellon university.
// MLOps Swag/Merch
https://shop.mlops.community/
// Related Links
Website: https://www.intuit.com/
--------------- ✌️Connect With Us ✌️ -------------
Join our slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register
Catch all episodes, blogs, newsletters, and more: https://mlops.community/
Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with Jineet on LinkedIn: https://www.linkedin.com/in/jineetdoshi/
MLOps.community