Evaluation Panel // Large Language Models in Production Conference Part II

12 snips

Aug 25, 2023

Guest

Amrutha Gujjar

Language model interpretability experts and AI researchers discuss challenges of evaluating large language models, the impact of chat GPT in the industry, evaluating model performance and data set quality, the use of large language models in machine learning, and tool sets, guardrails, and challenges in language models.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

INSIGHT

Evaluation Starts With Desired Behavior

Generative LLM projects rarely start by building a formal dataset; they begin with desired behavior and prompt design.
This shifts evaluation challenges to choosing the right test data and metrics rather than held-out training sets.

ADVICE

Measure Instruction Following And Safety

Measure both whether the model follows instructions and whether its answers are correct and coherent.
Actively evaluate hallucinations, context retention, safety, and latency as core metrics.

INSIGHT

Good Answers Are User-Dependent

'Good' answers vary by user expectations across tone, length, and conciseness.
Evaluation can and should be personalized to reflect those attribute preferences.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

MLOps Coffee Sessions #174 with Evaluation Panel, Amrutha Gujjar, Josh Tobin, and Sohini Roy hosted by Abi Aryan. We are now accepting talk proposals for our next LLM in Production virtual conference on October 3rd. Apply to speak here: https://go.mlops.community/NSAX1O // Abstract Language models are very complex thus introducing several challenges in interpretability. The large amounts of data required to train these black-box language models make it even harder to understand why a language model generates a particular output. In the past, transformer models were typically evaluated using perplexity, BLEU score, or human evaluation. However, LLMs amplify the problem even further due to their generative nature thus making them further susceptible to hallucinations and factual inaccuracies. Thus, evaluation becomes an important concern. // Bio Abi Aryan Machine Learning Engineer @ Independent Consultant Abi is a machine learning engineer and an independent consultant with over 7 years of experience in the industry using ML research and adapting it to solve real-world engineering challenges for businesses for a wide range of companies ranging from e-commerce, insurance, education and media & entertainment where she is responsible for machine learning infrastructure design and model development, integration and deployment at scale for data analysis, computer vision, audio-speech synthesis as well as natural language processing. She is also currently writing and working in autonomous agents and evaluation frameworks for large language models as a researcher at Bolkay. Amrutha Gujjar CEO & Co-Founder @ Structured Amrutha Gujjar is a senior software engineer and CEO and co-founder of Structured, based in New York. With a Bachelor of Science in Computer Science from the University of Washington's Allen School of CSE, she brings expertise in software development and leadership to my work.

Connect with Amrutha on LinkedIn to learn more about her experience and discuss exciting opportunities in software development and leadership.

Josh Tobin

Founder @ GantryJosh Tobin is the founder and CEO of Gantry. Previously, Josh worked as a deep learning & robotics researcher at OpenAI and as a management consultant at McKinsey. He is also the creator of Full Stack Deep Learning (fullstackdeeplearning.com), the first course focused on the emerging engineering discipline of production machine learning. Josh did his PhD in Computer Science at UC Berkeley advised by Pieter Abbeel.

Sohini Roy

Senior Developer Relations Manager @ NVIDIASohini Bianka Roy is a senior developer relations manager at NVIDIA, working within the Enterprise Product group. With a passion for the intersection of machine learning and operations, Sohini specializes in the domains of MLOps and LLMOps. With her extensive experience in the field, she plays a crucial role in bridging the gap between developers and enterprise customers, ensuring smooth integration and deployment of NVIDIA's cutting-edge technologies. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links ⁠ --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Abi on LinkedIn: https://www.linkedin.com/in/abiaryan/ Connect with Amrutha on LinkedIn: https://www.linkedin.com/in/amruthagujjar/ Connect with Josh on LinkedIn: https://www.linkedin.com/in/josh-tobin-4b3b10a9/ Connect with Sohini on Twitter: https://twitter.com/biankaroy_