MLOps.community

In this podcast, Manjot Pahwa, Rahul Parundekar, and Patrick Barker discuss the integration of Kubernetes and large language models (LLMs), the challenges of using Kubernetes for data scientists, and the considerations for hosting LMM applications in production. They also explore the abstraction of LLMs on Kubernetes, the cost considerations, and the pros and cons of using Kubernetes for LLM training versus inferencing. Additionally, they touch on using Kubernetes for real-time online inferences and the availability of abstractions like Metaplow.

Michelle Marie Conway, a tech industry professional, shares insights on continuous learning, gender diversity in STEM, and the potential of AI tools in MLOps. Topics include staying up to date with documentation, understanding code logic, challenges and benefits of AI tools, and the importance of communication with stakeholders in the banking sector. The importance of diversity in the tech industry and efforts to create inclusive environments are also discussed.

In this podcast, the MLOps vs. LLMOps Panel discuss the high-level differences between MLOps and LLMOps, the impact of ML ops on companies, the challenges of open source tools and data safety in financial firms, the cost and rationalization of MLOps, options for large enterprises in ML model development, and the use of foundational models and vector databases.

Beyang Liu, developer of Cody, an open-source AI coding assistant, discusses the challenges and process of incorporating AI into existing products, navigating complex code bases, and the technology used in building Cody. The chapter also touches on the complexity of fine-tuning AI models and supporting multiple language models in Cody.

Language model interpretability experts and AI researchers discuss challenges of evaluating large language models, the impact of chat GPT in the industry, evaluating model performance and data set quality, the use of large language models in machine learning, and tool sets, guardrails, and challenges in language models.

Lingjiao Chen discusses strategies to reduce the cost of using large language models (LLMs) and introduces FrugalGPT, which can match the performance of GPT-4 with up to 98% cost reduction. The podcast also explores optimizing LLM prompts, comparing API providers for cost and quality, approximating performance with a cache layer, and reducing the cost of using LLMs

Panelists George Mathew, Asmitha Rathis, Natalia Burina, and Sahar Mor discuss building products with LLMs, emphasizing transparency, control, and explainability. They explore the challenges of prompting in language models and provide tips for avoiding impersonation and hallucination. They highlight the importance of feedback loops in improving language models and discuss the economic components of using APIs and inference calls. The panel concludes with excitement about the conference and promotion of their own podcast.

Thibaut Labarre, AngelList investing and natural language processing expert, discusses the innovative use of large language models at AngelList, including news article classification for investor dashboards. They also talk about the challenges of prompt engineering, the importance of involving domain experts, and the ethical concerns of using AI models for reading legal texts.

MLOps Coffee Sessions #170 with Phillip Carter, All the Hard Stuff with LLMs in Product Development. We are now accepting talk proposals for our next LLM in Production virtual conference on October 3rd. Apply to speak here: https://go.mlops.community/NSAX1O // Abstract Delve into challenges in implementing LLMs, such as security concerns and collaborative measures against attacks. Emphasize the role of ML engineers and product managers in successful implementation. Explore identifying leading indicators and measuring ROI for impactful AI initiatives. // Bio Phillip is on the product team at Honeycomb where he works on a bunch of different developer tooling things. He's an OpenTelemetry maintainer -- chances are if you've read the docs to learn how to use OTel, you've read his words. He's also Honeycomb's (accidental) prompt engineering expert by virtue of building and shipping products that use LLMs. In a past life, he worked on developer tools at Microsoft, helping bring the first cross-platform version of .NET into the world and grow to 5 million active developers. When not doing computer stuff, you'll find Phillip in the mountains riding a snowboard or backpacking in the Cascades. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links ⁠Website: https://phillipcarter.dev/ https://www.honeycomb.io/blog/improving-llms-production-observability https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llm https://phillipcarter.dev/posts/how-to-make-an-fsharp-code-fixer/ The "hard stuff" post: https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llm Our follow-up on iterating on LLMs in prod: https://www.honeycomb.io/blog/improving-llms-production-observability --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Phillip on LinkedIn: https://www.linkedin.com/in/phillip-carter-4714a135/ Timestamps: [00:00] Phillip's preferred coffee [00:33] Takeaways [01:53] Please like, share, and subscribe to our MLOps channels! [02:45] Phillip's background [07:15] Querying Natural Language [11:25] Function calls [14:29] Pasting errors or traces [16:30] Error patterns [20:22] Honeycomb's Improvement cycle [23:20] Prompt boxes rationale [28:06] Prompt injection cycles [32:11] Injection Attempt [33:30] UI undervalued, charging the AI feature [35:11] ROI cost [44:26] Bridging ML and Product Perspective [52:53] AI Model Trade-offs [56:33] Query assistant [59:07] Honeycomb is hiring! [1:00:08] Wrap up

MLOps Coffee Sessions #169 with Barak Turovsky, MLOps at the Age of Generative AI. Thanks to wandb.ai for sponsoring this episode. Check out their new course on evaluating and fine-tuning LLMs wandb.me/genai-mlops.course // Abstract The talk focuses on MLOps aspects of developing, training and serving Generative AI/Large Language models // Bio Barak is an Executive in Residence at Scale Venture Partners, a leading Enterprise venture capital firm. Barak spent 10 years as Head of Product and User Experience for Languages AI and Google Translate teams within the Google AI org, focusing on applying cutting-edge Artificial Intelligence and Machine Learning technologies to deliver magical experiences across Google Search, Assistant, Cloud, Chrome, Ads, and other products. Previously, Barak spent 2 years as a product leader within the Google Commerce team. Most recently, Barak served as Chief Product Officer, responsible for product management and engineering at Trax, a leading provider of Computer Vision AI solutions for Retail and Commerce industries. Prior to joining Google in 2011, Barak was Director of Products in Microsoft’s Mobile Advertising, Head of Mobile Commerce at PayPal, and Chief Technical Officer at an Israeli start-up. He lived more than 10 years in 3 different countries (Russia, Israel, and the US) and fluently speaks three languages. Barak earned a Bachelor of Laws degree from Tel Aviv University, Israel, and a Master’s of Business Administration from the University of California, Berkeley. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links Bio and links about Barak's work: https://docs.google.com/document/d/1E4Yrmt_Y57oTEYHQQDvt71XzSJ8Ew5WvscAQbHV4K3U/edit Framework for evaluating Generative AI use cases: https://www.linkedin.com/pulse/framework-evaluating-generative-ai-use-cases-barak-turovsky/?trackingId=%2BMRxEZ9WTPCNH2JscILTeg%3D%3D The Great A.I. Awakening: https://www.nytimes.com/2016/12/14/magazine/the-great-ai-awakening.html --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Barak on LinkedIn: https://www.linkedin.com/in/baraktur/ Timestamps: [00:00] Barak's preferred coffee [00:23] Barak Turovsky's background [03:10] Please like, share, and subscribe to our MLOps channels! [04:09] Getting into tech [08:39] First wave of AI [12:39] Building a product at a scale and the challenges [15:59] Framework for evaluating Generative AI use cases [24:33] Machine trust adoption [29:13] Wandb's new course [31:10] Focus on achievable use cases for LLMs. [36:36] User feedback [38:23] Disruption of entertainment and customer interactions [46:14] Get new tools or work with your own distribution? [47:57] Importance of data engineers [53:28] ML Engineers Collaborate with Product [56:13] Wrap up