MLOps.community

Language model interpretability experts and AI researchers discuss challenges of evaluating large language models, the impact of chat GPT in the industry, evaluating model performance and data set quality, the use of large language models in machine learning, and tool sets, guardrails, and challenges in language models.

Lingjiao Chen discusses strategies to reduce the cost of using large language models (LLMs) and introduces FrugalGPT, which can match the performance of GPT-4 with up to 98% cost reduction. The podcast also explores optimizing LLM prompts, comparing API providers for cost and quality, approximating performance with a cache layer, and reducing the cost of using LLMs

Panelists George Mathew, Asmitha Rathis, Natalia Burina, and Sahar Mor discuss building products with LLMs, emphasizing transparency, control, and explainability. They explore the challenges of prompting in language models and provide tips for avoiding impersonation and hallucination. They highlight the importance of feedback loops in improving language models and discuss the economic components of using APIs and inference calls. The panel concludes with excitement about the conference and promotion of their own podcast.

Thibaut Labarre, AngelList investing and natural language processing expert, discusses the innovative use of large language models at AngelList, including news article classification for investor dashboards. They also talk about the challenges of prompt engineering, the importance of involving domain experts, and the ethical concerns of using AI models for reading legal texts.

MLOps Coffee Sessions #170 with Phillip Carter, All the Hard Stuff with LLMs in Product Development. We are now accepting talk proposals for our next LLM in Production virtual conference on October 3rd. Apply to speak here: https://go.mlops.community/NSAX1O// AbstractDelve into challenges in implementing LLMs, such as security concerns and collaborative measures against attacks. Emphasize the role of ML engineers and product managers in successful implementation. Explore identifying leading indicators and measuring ROI for impactful AI initiatives.// BioPhillip is on the product team at Honeycomb, where he works on a bunch of different developer tooling things. He's an OpenTelemetry maintainer -- chances are, if you've read the docs to learn how to use OTel, you've read his words. He's also Honeycomb's (accidental) prompt engineering expert by virtue of building and shipping products that use LLMs. In a past life, he worked on developer tools at Microsoft, helping bring the first cross-platform version of .NET into the world and grow to 5 million active developers. When not doing computer stuff, you'll find Phillip in the mountains riding a snowboard or backpacking in the Cascades.// MLOps Jobs board jobs.mlops.community// MLOps Swag/Merchhttps://mlops-community.myshopify.com/// Related Links⁠Website: https://phillipcarter.dev/https://www.honeycomb.io/blog/improving-llms-production-observabilityhttps://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llmhttps://phillipcarter.dev/posts/how-to-make-an-fsharp-code-fixer/The "hard stuff" post: https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llmOur follow-up on iterating on LLMs in prod: https://www.honeycomb.io/blog/improving-llms-production-observability--------------- ✌️Connect With Us ✌️ -------------Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerCatch all episodes, blogs, newsletters, and more: https://mlops.community/Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with Phillip on LinkedIn: https://www.linkedin.com/in/phillip-carter-4714a135/Timestamps:[00:00] Phillip's preferred coffee[00:33] Takeaways[01:53] Please like, share, and subscribe to our MLOps channels![02:45] Phillip's background[07:15] Querying Natural Language[11:25] Function calls[14:29] Pasting errors or traces[16:30] Error patterns[20:22] Honeycomb's Improvement cycle[23:20] Prompt boxes rationale[28:06] Prompt injection cycles[32:11] Injection Attempt[33:30] UI undervalued, charging the AI feature[35:11] ROI cost[44:26] Bridging ML and Product Perspective[52:53] AI Model Trade-offs[56:33] Query assistant[59:07] Honeycomb is hiring![1:00:08] Wrap up

MLOps Coffee Sessions #169 with Barak Turovsky, MLOps at the Age of Generative AI.Thanks to Weights & Biases for sponsoring this episode. Check out their new course on evaluating and fine-tuning LLMs at wandb.me/genai-mlops.course// AbstractThe talk focuses on MLOps aspects of developing, training, and serving Generative AI/Large Language models// BioBarak is an Executive in Residence at Scale Venture Partners, a leading Enterprise venture capital firm. Barak spent 10 years as Head of Product and User Experience for Languages AI and Google Translate teams within the Google AI org, focusing on applying cutting-edge Artificial Intelligence and Machine Learning technologies to deliver magical experiences across Google Search, Assistant, Cloud, Chrome, Ads, and other products. Previously, Barak spent 2 years as a product leader within the Google Commerce team. Most recently, Barak served as Chief Product Officer, responsible for product management and engineering at Trax, a leading provider of Computer Vision AI solutions for the Retail and Commerce industries. Prior to joining Google in 2011, Barak was Director of Products in Microsoft’s Mobile Advertising, Head of Mobile Commerce at PayPal, and Chief Technical Officer at an Israeli start-up. He lived more than 10 years in 3 different countries (Russia, Israel, and the US) and speaks three languages. Barak earned a Bachelor of Laws degree from Tel Aviv University, Israel, and a Master’s of Business Administration from the University of California, Berkeley.// MLOps Jobs board jobs.mlops.community// MLOps Swag/Merchhttps://mlops-community.myshopify.com/// Related LinksBio and links about Barak's work: https://docs.google.com/document/d/1E4Yrmt_Y57oTEYHQQDvt71XzSJ8Ew5WvscAQbHV4K3U/editFramework for evaluating Generative AI use cases: https://www.linkedin.com/pulse/framework-evaluating-generative-ai-use-cases-barak-turovsky/?trackingId=%2BMRxEZ9WTPCNH2JscILTeg%3D%3DThe Great A.I. Awakening: https://www.nytimes.com/2016/12/14/magazine/the-great-ai-awakening.html--------------- ✌️Connect With Us ✌️ -------------Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerCatch all episodes, blogs, newsletters, and more: https://mlops.community/Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with Barak on LinkedIn: https://www.linkedin.com/in/baraktur/Timestamps:[00:00] Barak's preferred coffee[00:23] Barak Turovsky's background[03:10] Please like, share, and subscribe to our MLOps channels![04:09] Getting into tech[08:39] First wave of AI[12:39] Building a product at scale and the challenges[15:59] Framework for evaluating Generative AI use cases[24:33] Machine trust adoption[29:13] Wandb's new course[31:10] Focus on achievable use cases for LLMs[36:36] User feedback[38:23] Disruption of entertainment and customer interactions [46:14] Get new tools or work with your own distribution?[47:57] Importance of data engineers[53:28] ML Engineers Collaborate with Product[56:13] Wrap up

MLOps Coffee Sessions #168 with Piotr Niedźwiedź, Experiment Tracking in the Age of LLMs, co-hosted by Vishnu Rachakonda.// AbstractPiotr shares his journey as an entrepreneur and the importance of focusing on core values to achieve success. He highlights the mission of Neptune to support ML teams by providing them with control and confidence in their models. The conversation delves into the role of experiment tracking in understanding and debugging models, comparing experiments, and versioning models. Piotr introduces the concept of prompt engineering as a different approach to building models, emphasizing the need for prompt validation and testing methods.// BioPiotr is the CEO of neptune.ai. Day to day, apart from running the company, he focuses on the product side of things. Strategy, planning, ideation, getting deep into user needs and use cases. He really likes it. Piotr's path to ML started with software engineering. Always liked math and started programming when he was 7. In high school, Piotr got into algorithmics and programming competitions and loved competing with the best. That got him into the best CS and Maths program in Poland which funny enough today specializes in machine learning. Piotr did his internships at Facebook and Google and was offered to stay in the Valley. But something about being a FAANG engineer didn’t feel right. He had this spark to do more, build something himself. So with a few of his friends from the algo days, they started Codilime, a software consultancy, and later a sister company Deepsense.ai machine learning consultancy, where he was a CTO. When he came to the ML space from software engineering, he was surprised by the messy experimentation practices, lack of control over model building, and a missing ecosystem of tools to help people deliver models confidently. It was a stark contrast to the software development ecosystem, where you have mature tools for DevOps, observability, or orchestration to execute efficiently in production. And then, one day, some ML engineers from Deepsense.ai came to him and showed him this tool for tracking experiments they built during a Kaggle competition (which they won btw), and he knew this could be big. He asked around, and everyone was struggling with managing experiments. He decided to spin it off as a VC-funded product company, and the rest is history.// MLOps Jobs board jobs.mlops.community// MLOps Swag/Merchhttps://mlops-community.myshopify.com/// Related Linkshttps://neptune.ai/blog/author/piotr-niedzwiedzhttps://www.youtube.com/playlist?list=PLKePQLVx9tOfKFbg9GY2Anl41Be4T1-m5 https://thesequence.substack.com/p/-piotr-niedzwiedz-neptunes-ceo-onhttps://open.spotify.com/episode/2KEqTMAHODbPKdUEtlrhm7?si=ed862b2ac7534e39https://www.linkedin.com/in/piotrniedzwiedz/--------------- ✌️Connect With Us ✌️ -------------Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerCatch all episodes, blogs, newsletters, and more: https://mlops.community/Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with Piotr on LinkedIn: https://www.linkedin.com/in/piotrniedzwiedz/ Timestamps:[00:00] Introduction to Piotr Niedźwiedź[01:35] Please like, share, and subscribe to our MLOps channels![01:58] Wojciech Zaremba[05:20] The Olympiad[06:31] Building own company[12:16] Talking outside Poland with the same passion[13:45] Adapting with Neptune[19:35] Core values focus[22:02] Product Vision changes with advances[29:36] Control and confidence[30:05] Experiment tracking existing use cases[37:25] Control pane[38:59] Piotr's prediction[43:20] WiFi issues around the world[44:09] Wrap up

MLOps Coffee Sessions #167 with Maxime Beauchemin, Treating Prompt Engineering More Like Code.// AbstractPromptimize is an innovative tool designed to scientifically evaluate the effectiveness of prompts. Discover the advantages of open-sourcing the tool and its relevance, drawing parallels with test suites in software engineering. Uncover the increasing interest in this domain and the necessity for transparent interactions with language models. Delve into the world of prompt optimization, deterministic evaluation, and the unique challenges in AI prompt engineering. // BioMaxime Beauchemin is the founder and CEO of Preset, a Series B startup supporting and commercializing the Apache Superset project. Max was the original creator of Apache Airflow and Apache Superset when he was at Airbnb. Max has over a decade of experience in data engineering at companies like Lyft, Airbnb, Facebook, and Ubisoft.// MLOps Jobs board jobs.mlops.community// MLOps Swag/Merchhttps://mlops-community.myshopify.com/// Related LinksMax's first MLOps Podcast episode: https://go.mlops.community/KBnOgNTest-Driven Prompt Engineering for LLMs with Promptimize blog: https://maximebeauchemin.medium.com/mastering-ai-powered-product-development-introducing-promptimize-for-test-driven-prompt-bffbbca91535https://maximebeauchemin.medium.com/mastering-ai-powered-product-development-Test-Driven Prompt Engineering for LLMs with Promptimize podcast: https://talkpython.fm/episodes/show/417/test-driven-prompt-engineering-for-llms-with-promptimizeTaming AI Product Development Through Test-driven Prompt Engineering // Maxime Beauchemin // LLMs in Production Conference lightning talk: https://home.mlops.community/home/videos/taming-ai-product-development-through-test-driven-prompt-engineering--------------- ✌️Connect With Us ✌️ -------------Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerCatch all episodes, blogs, newsletters, and more: https://mlops.community/Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with Max on LinkedIn: https://www.linkedin.com/in/maximebeauchemin/Timestamps:[00:00] Max introduces the Apache Superset project at Preset[01:04] Max's preferred coffee[01:16] Airflow creator[01:45] Takeaways[03:53] Please like, share, and subscribe to our MLOps channels![04:31] Check Max's first MLOps Podcast episode[05:20] Promptimize[06:10] Interaction with API[08:27] Deterministic evaluation of SQL queries and AI[12:40] Figuring out the right edge cases[14:17] Reaction with Vector Database[15:55] Promptomize Test Suite[18:48] Promptimize vision[20:47] The open-source blood[23:04] Impact of open source[23:18] Dangers of open source[25:25] AI-Language Models Revolution[27:36] Test-driven design[29:46] Prompt tracking[33:41] Building Test Suites as Assets[36:49] Adding new prompt cases to new capabilities[39:32] Monitoring speed and cost[44:07] Creating own benchmarks[46:19] AI feature adding more value to the end users[49:39] Perceived value of the feature[50:53] LLMs costs[52:15] Specialized model versus Generalized model[56:58] Fine-tuning LLMs use cases[1:02:30] Classic Engineer's Dilemma[1:03:46] Build exciting tech that's available[1:05:02] Catastrophic forgetting[1:10:28] Prompt-driven development[1:13:23] Wrap up

MLOps Coffee Sessions #166 with Roy Hasson & Santona Tuli, Eliminating Garbage In/Garbage Out for Analytics and ML.// AbstractShift left data quality ownership and observability that makes it easy for users to catch bad data at the source and stop it from entering your analytics/ML stack.// BioSantona TuliSantona Tuli, Ph.D., began her data journey through fundamental physics—searching through massive event data from particle collisions at CERN to detect rare particles. She’s since extended her machine learning engineering to natural language processing, before switching focus to product and data engineering for data workflow authoring frameworks. As a Python engineer, she started with the programmatic data orchestration tool, Airflow, helping improve its developer experience for data science and machine learning pipelines. Currently, at Upsolver, she leads data engineering and science, driving developer research and engagement for the declarative workflow authoring framework in SQL. Dr. Tuli is passionate about building, as well as empowering others to build, end-to-end data and ML pipelines, scalably. Roy HassonRoy is the head of product at Upsolver, helping companies deliver high-quality data to their analytics and ML tools. Previously, Roy led product management for AWS Glue and AWS Lake Formation.// MLOps Jobs board jobs.mlops.community// MLOps Swag/Merchhttps://mlops-community.myshopify.com/// Related Linkshttps://royondata.substack.com/--------------- ✌️Connect With Us ✌️ -------------Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup:https://go.mlops.community/registerCatch all episodes, blogs, newsletters, and more: https://mlops.community/Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with Roy on LinkedIn: https://www.linkedin.com/in/royhasson/Connect with Santona on LinkedIn: https://www.linkedin.com/in/santona-tuli/Timestamps:[00:00] Santona's and Roy's preferred coffee[01:05] Santona's and Roy's background[03:33] Takeaways[05:49] Please like, share, and subscribe to our MLOps channels![06:42] Back story of having Santona and Roy on the podcast[09:51] Santona's story[11:37] Optimal tag teamwork[16:53] Dealing with stakeholder needs[26:25] Having mechanisms in place[27:30] Building for data Engineers vs building for data scientists[34:50] Creating solutions for users[38:55] User experience holistic point of view[41:11] Tooling sprawl is real[42:00] LLMs' reliability [45:00] Things I would have loved to learn five years ago[49:46] Wrap up

MLOps Coffee Sessions #165 with Sammy Sidhu, Python Power: How Daft Embeds Models and Revolutionizes Data Processing.// AbstractSammy shares his fascinating journey in the autonomous vehicle industry, highlighting his involvement in two successful startup acquisitions by Tesla and Toyota. He emphasizes his expertise in optimizing and distilling models for efficient machine learning, which he has incorporated into his new company, Eventual. The company's open-source offering, daf, focuses on tackling the challenges of unstructured and complex data. Sammy discusses the future of MLOps, machine learning, and data storage, particularly in relation to the retrieval and processing of unstructured data. The Eventual team is developing Daft, an open-source query engine that aims to provide efficient data storage solutions for unstructured data, offering features like governance, schema evolution, and time travel. The conversation sheds light on the innovative developments in the field and the potential impact on various industries.// BioSammy is a Deep Learning and systems veteran, holding over a dozen publications and patents in the space. Sammy graduated from the University of California, Berkeley, where he did research in Deep Learning and High Performance Computing. He then joined DeepScale as the Chief Architect and led the development of perception technologies for autonomous vehicles. During this time, DeepScale grew rapidly and was subsequently acquired by Tesla in 2019. Staying in Autonomous Vehicles, Sammy joined Lyft Level 5 as a Senior Staff Software Engineer, building out core perception algorithms as well as infrastructure for machine learning and embedded systems. Level 5 was then acquired by Toyota in 2021, adopting much of its work. Sammy is now CEO and Co-Founder at Eventual Building Daft, an open-source query engine that specializes in multimodal data. // MLOps Jobs board jobs.mlops.community// MLOps Swag/Merchhttps://mlops-community.myshopify.com/// Related Linkshttps://sammysidhu.com/Check out Daft, our open-source query engine for multimodal data! https://www.getdaft.io/Here are some talks/shows we have given about it:- PyData Global (Dec 2022): Large-scale image processing: https://www.youtube.com/watch?v=ol6IQUbyeDo&ab_channel=PyData- Ray Meetup (March 2023): Distributed ML preprocessing + training on Ray https://www.youtube.com/watch?v=1MpEYlIlu7w&t=2972s&ab_channel=Anyscale- The Data Stack Show (April 2023): Self-Driving Technology and Data Infrastructure with Sammy Sidhu https://datastackshow.com/podcast/the-prql-self-driving-technology-and-data-infrastructure-with-sammy-sidhu-co-founder-and-ceo-of-eventual/Chain of Thought for LLMs: https://cobusgreyling.medium.com/chain-of-thought-prompting-in-llms-1077164edf97Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes: https://arxiv.org/abs/2305.02301--------------- ✌️Connect With Us ✌️ -------------Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerCatch all episodes, blogs, newsletters, and more: https://mlops.community/Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with Sammy on LinkedIn: https://www.linkedin.com/in/sammy-sidhu/