Measuring Chatbot Effectiveness

Chinar Movsisyan is the co-founder and CEO of Feedback Intelligence (formerly Manot), an MLOps startup based in San Francisco. She has been in the AI field for more than 7 years from research labs to venture-backed startups.

Reliable LLM Products, Fueled by Feedback // MLOps Podcast #250 with Chinar Movsisyan, CEO of Feedback Intelligence. // Abstract We live in a world driven by large language models (LLMs) and generative AI, but ensuring they are ready for real-world deployment is crucial. Despite the availability of numerous evaluation tools, many LLM products still struggle to make it to production. We propose a new perspective on how LLM products should be measured, evaluated, and improved. A product is only as good as the user's experience and expectations, and we aim to enhance LLM products to meet these standards reliably. Our approach creates a new category that automates the need for separate evaluation, observability, monitoring, and experimentation tools. By starting with the user experience and working backward to the model, we provide a comprehensive view of how the product is actually used, rather than how it is intended to be used. This user-centric aka feedback-centric approach is the key to every successful product. // Bio Chinar Movsisyan is the founder and CEO of Feedback Intelligence, an MLOps company based in San Francisco that enables enterprises to make sure that LLM-based products are reliable and that the output is aligned with end-user expectations. With over eight years of experience in deep learning, spanning from research labs to venture-backed startups, Chinar has led AI projects in mission-critical applications such as healthcare, drones, and satellites. Her primary research interests include artificial intelligence, generative AI, machine learning, deep learning, and computer vision. At Feedback Intelligence, Chinar and her team address a crucial challenge in LLM development by automatically converting user feedback into actionable insights, enabling AI teams to analyze root causes, prioritize issues, and accelerate product optimization. This approach is particularly valuable in highly regulated industries, helping enterprises to reduce time-to-market and time-to-resolution while ensuring robust LLM products. Feedback Intelligence, which participated in the Berkeley SkyDeck accelerator program, is currently expanding its business across various verticals. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links Website: https://www.manot.ai/ --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Chinar on LinkedIn: https://www.linkedin.com/in/nik-suresh/ Timestamps: [00:00] Chinar's preferred coffee [00:20] Takeaways [02:25] Please like, share, leave a review, and subscribe to our MLOps channels! [03:23] Object Detection on Drones [06:10] Street Surveillance Detection Use Case [08:00] Optimizing Vision Models [09:50] Data Engineering for AI/ML Conference Ad [10:42] Plastic surgery project [12:33] Diffusion models getting popular [13:57] AI challenges in highly regulated industries [17:48] Product metrics evaluation insights [20:55] Chatbot effectiveness metrics [23:15] Interpreting user signals [24:45] Metadata tracking in LLM [27:41] Agentic workflow [28:53] Effective data analysis strategies [30:41] Identifying key metrics [33:59] AI metrics role shift [37:20] Tooling for non-engineers [42:12] Balancing engineering and evaluation [44:39] Bridging SME engineering gap [46:41] Expand expertise potential [47:40] What's with flamingos [48:04] Wrap up