Evolution of Evaluation Sets and Data Infrastructure in Machine Learning

Join us at our first in-person conference on June 25 all about AI Quality: https://www.aiqualityconference.com/ // Abstract Dive into the challenges of scaling AI models from Minimum Viable Product (MVP) to full production. The panel emphasizes the importance of continually updating knowledge and data, citing examples like teaching AI systems nuanced concepts and handling brand name translations.

User feedback's role in model training, alongside evaluation steps like human annotation and heuristic-based assessment, was highlighted.

The speakers stressed the necessity of tooling for user evaluation, version control, and regular performance updates. Insights on in-house and external tools for annotation and evaluation were shared, providing a comprehensive view of the complexities involved in scaling AI models. // Bio Alex Volkov - Moderator Alex Volkov is an AI Evangelist at Weights & Biases, celebrated for his expertise in clarifying the complexities of AI and advocating for its beneficial uses. He is the founder and host of ThursdAI, a weekly newsletter, and podcast that explores the latest in AI, its practical applications, open-source, and innovation. With a solid foundation as an AI startup founder and 20 years in full-stack software engineering, Alex offers a deep well of experience and insight into AI innovation. Eric Peter Product management leader and 2x founder with experience in enterprise products, data, and machine learning. Currently building tools for generative AI @Databricks. Donné Stevenson Focused on building AI-powered products that give companies the tools and expertise needed to harness to power of AI in their respective fields. Phillip Carter Phillip is on the product team at Honeycomb where he works on a bunch of different developer tooling things. He's an OpenTelemetry maintainer -- chances are if you've read the docs to learn how to use OTel, you've read his words. He's also Honeycomb's (accidental) prompt engineering expert by virtue of building and shipping products that use LLMs. In a past life, he worked on developer tools at Microsoft, helping bring the first cross-platform version of .NET into the world and grow to 5 million active developers. When not doing computer stuff, you'll find Phillip in the mountains riding a snowboard or backpacking in the Cascades. Andrew Hoh Andrew Hoh is the President and Co-Founder of LastMile AI. Previously, he was a Group PM Manager at Meta AI, driving product for their AI Platform. Previously, he was the Product Manager for the Machine Learning Infrastructure team at Airbnb and a founding team member of Azure Cosmos DB, Microsoft Azure's distributed NoSQL database. He graduated with a BA in Computer Science from Dartmouth College. A big thank you to our Premium Sponsors,  @Databricks  and  @baseten  for their generous support! // Sign up for our Newsletter to never miss an event: https://mlops.community/join/ // Watch all the conference videos here: https://home.mlops.community/home/collections // Check out the MLOps Community podcast: https://open.spotify.com/show/7wZygk3mUUqBaRbBGB1lgh?si=242d3b9675654a69 // Read our blog: mlops.community/blog // Join an in-person local meetup near you: https://mlops.community/meetups/ // MLOps Swag/Merch: https://mlops-community.myshopify.com/ // Follow us on Twitter: https://twitter.com/mlopscommunity //Follow us on Linkedin: https://www.linkedin.com/company/mlopscommunity/