MLOps.community

Lingjiao Chen discusses strategies to reduce the cost of using large language models (LLMs) and introduces FrugalGPT, which can match the performance of GPT-4 with up to 98% cost reduction. The podcast also explores optimizing LLM prompts, comparing API providers for cost and quality, approximating performance with a cache layer, and reducing the cost of using LLMs

Panelists George Mathew, Asmitha Rathis, Natalia Burina, and Sahar Mor discuss building products with LLMs, emphasizing transparency, control, and explainability. They explore the challenges of prompting in language models and provide tips for avoiding impersonation and hallucination. They highlight the importance of feedback loops in improving language models and discuss the economic components of using APIs and inference calls. The panel concludes with excitement about the conference and promotion of their own podcast.

Thibaut Labarre, AngelList investing and natural language processing expert, discusses the innovative use of large language models at AngelList, including news article classification for investor dashboards. They also talk about the challenges of prompt engineering, the importance of involving domain experts, and the ethical concerns of using AI models for reading legal texts.

MLOps Coffee Sessions #170 with Phillip Carter, All the Hard Stuff with LLMs in Product Development. We are now accepting talk proposals for our next LLM in Production virtual conference on October 3rd. Apply to speak here: https://go.mlops.community/NSAX1O // Abstract Delve into challenges in implementing LLMs, such as security concerns and collaborative measures against attacks. Emphasize the role of ML engineers and product managers in successful implementation. Explore identifying leading indicators and measuring ROI for impactful AI initiatives. // Bio Phillip is on the product team at Honeycomb where he works on a bunch of different developer tooling things. He's an OpenTelemetry maintainer -- chances are if you've read the docs to learn how to use OTel, you've read his words. He's also Honeycomb's (accidental) prompt engineering expert by virtue of building and shipping products that use LLMs. In a past life, he worked on developer tools at Microsoft, helping bring the first cross-platform version of .NET into the world and grow to 5 million active developers. When not doing computer stuff, you'll find Phillip in the mountains riding a snowboard or backpacking in the Cascades. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links ⁠Website: https://phillipcarter.dev/ https://www.honeycomb.io/blog/improving-llms-production-observability https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llm https://phillipcarter.dev/posts/how-to-make-an-fsharp-code-fixer/ The "hard stuff" post: https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llm Our follow-up on iterating on LLMs in prod: https://www.honeycomb.io/blog/improving-llms-production-observability --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Phillip on LinkedIn: https://www.linkedin.com/in/phillip-carter-4714a135/ Timestamps: [00:00] Phillip's preferred coffee [00:33] Takeaways [01:53] Please like, share, and subscribe to our MLOps channels! [02:45] Phillip's background [07:15] Querying Natural Language [11:25] Function calls [14:29] Pasting errors or traces [16:30] Error patterns [20:22] Honeycomb's Improvement cycle [23:20] Prompt boxes rationale [28:06] Prompt injection cycles [32:11] Injection Attempt [33:30] UI undervalued, charging the AI feature [35:11] ROI cost [44:26] Bridging ML and Product Perspective [52:53] AI Model Trade-offs [56:33] Query assistant [59:07] Honeycomb is hiring! [1:00:08] Wrap up

MLOps Coffee Sessions #169 with Barak Turovsky, MLOps at the Age of Generative AI. Thanks to wandb.ai for sponsoring this episode. Check out their new course on evaluating and fine-tuning LLMs wandb.me/genai-mlops.course // Abstract The talk focuses on MLOps aspects of developing, training and serving Generative AI/Large Language models // Bio Barak is an Executive in Residence at Scale Venture Partners, a leading Enterprise venture capital firm. Barak spent 10 years as Head of Product and User Experience for Languages AI and Google Translate teams within the Google AI org, focusing on applying cutting-edge Artificial Intelligence and Machine Learning technologies to deliver magical experiences across Google Search, Assistant, Cloud, Chrome, Ads, and other products. Previously, Barak spent 2 years as a product leader within the Google Commerce team. Most recently, Barak served as Chief Product Officer, responsible for product management and engineering at Trax, a leading provider of Computer Vision AI solutions for Retail and Commerce industries. Prior to joining Google in 2011, Barak was Director of Products in Microsoft’s Mobile Advertising, Head of Mobile Commerce at PayPal, and Chief Technical Officer at an Israeli start-up. He lived more than 10 years in 3 different countries (Russia, Israel, and the US) and fluently speaks three languages. Barak earned a Bachelor of Laws degree from Tel Aviv University, Israel, and a Master’s of Business Administration from the University of California, Berkeley. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links Bio and links about Barak's work: https://docs.google.com/document/d/1E4Yrmt_Y57oTEYHQQDvt71XzSJ8Ew5WvscAQbHV4K3U/edit Framework for evaluating Generative AI use cases: https://www.linkedin.com/pulse/framework-evaluating-generative-ai-use-cases-barak-turovsky/?trackingId=%2BMRxEZ9WTPCNH2JscILTeg%3D%3D The Great A.I. Awakening: https://www.nytimes.com/2016/12/14/magazine/the-great-ai-awakening.html --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Barak on LinkedIn: https://www.linkedin.com/in/baraktur/ Timestamps: [00:00] Barak's preferred coffee [00:23] Barak Turovsky's background [03:10] Please like, share, and subscribe to our MLOps channels! [04:09] Getting into tech [08:39] First wave of AI [12:39] Building a product at a scale and the challenges [15:59] Framework for evaluating Generative AI use cases [24:33] Machine trust adoption [29:13] Wandb's new course [31:10] Focus on achievable use cases for LLMs. [36:36] User feedback [38:23] Disruption of entertainment and customer interactions [46:14] Get new tools or work with your own distribution? [47:57] Importance of data engineers [53:28] ML Engineers Collaborate with Product [56:13] Wrap up

MLOps Coffee Sessions #168 with Piotr Niedźwiedź, Experiment Tracking in the Age of LLMs, co-hosted by Vishnu Rachakonda. // Abstract Piotr shares his journey as an entrepreneur and the importance of focusing on core values to achieve success. He highlights the mission of Neptune to support ML teams by providing them with control and confidence in their models. The conversation delves into the role of experiment tracking in understanding and debugging models, comparing experiments, and versioning models. Piotr introduces the concept of prompt engineering as a different approach to building models, emphasizing the need for prompt validation and testing methods. // Bio Piotr is the CEO of neptune.ai. Day to day, apart from running the company, he focuses on the product side of things. Strategy, planning, ideation, getting deep into user needs and use cases. He really likes it. Piotr's path to ML started with software engineering. Always liked math and started programming when he was 7. In high school, Piotr got into algorithmics and programming competitions and loved competing with the best. That got him into the best CS and Maths program in Poland which funny enough today specializes in machine learning. Piotr did his internships at Facebook and Google and was offered to stay in the Valley. But something about being a FAANG engineer didn’t feel right. He had this spark to do more, build something himself. So with a few of his friends from the algo days, they started Codilime, a software consultancy, and later a sister company Deepsense.ai machine learning consultancy, where he was a CTO. When he came to the ML space from software engineering, he was surprised by the messy experimentation practices, lack of control over model building, and a missing ecosystem of tools to help people deliver models confidently. It was a stark contrast to the software development ecosystem, where you have mature tools for DevOps, observability, or orchestration to execute efficiently in production. And then, one day, some ML engineers from Deepsense.ai came to him and showed him this tool for tracking experiments they built during a Kaggle competition (which they won btw), and he knew this could be big. He asked around, and everyone was struggling with managing experiments. He decided to spin it off as a VC-funded product company, and the rest is history. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links https://neptune.ai/blog/author/piotr-niedzwiedz https://www.youtube.com/playlist?list=PLKePQLVx9tOfKFbg9GY2Anl41Be4T1-m5 https://thesequence.substack.com/p/-piotr-niedzwiedz-neptunes-ceo-on https://open.spotify.com/episode/2KEqTMAHODbPKdUEtlrhm7?si=ed862b2ac7534e39https://www.linkedin.com/in/piotrniedzwiedz/ --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Piotr on LinkedIn: https://www.linkedin.com/in/piotrniedzwiedz/ Timestamps: [00:00] Introduction to Piotr Niedźwiedź [01:35] Please like, share, and subscribe to our MLOps channels! [01:58] Wojciech Zaremba [05:20] The Olympiad [06:31] Building own company [12:16] Talking outside Poland with the same passion [13:45] Adapting with Neptune [19:35] Core values focus [22:02] Product Vision changes with advances [29:36] Control and confidence [30:05] Experiment tracking existing use cases [37:25] Control pane [38:59] Piotr's prediction [43:20] WiFi issues around the world [44:09] Wrap up

MLOps Coffee Sessions #167 with Maxime Beauchemin, Treating Prompt Engineering More Like Code. // Abstract Promptimize is an innovative tool designed to scientifically evaluate the effectiveness of prompts. Discover the advantages of open-sourcing the tool and its relevance, drawing parallels with test suites in software engineering. Uncover the increasing interest in this domain and the necessity for transparent interactions with language models. Delve into the world of prompt optimization, deterministic evaluation, and the unique challenges in AI prompt engineering. // Bio Maxime Beauchemin is the founder and CEO of Preset, a series B startup supporting and commercializing the Apache Superset project. Max was the original creator of Apache Airflow and Apache Superset when he was at Airbnb. Max has over a decade of experience in data engineering, at companies like Lyft, Airbnb, Facebook, and Ubisoft. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links Max's first MLOps Podcast episode: https://go.mlops.community/KBnOgN Test-Driven Prompt Engineering for LLMs with Promptimize blog: https://maximebeauchemin.medium.com/mastering-ai-powered-product-development-introducing-promptimize-for-test-driven-prompt-bffbbca91535https://maximebeauchemin.medium.com/mastering-ai-powered-product-development-Test-Driven Prompt Engineering for LLMs with Promptimize podcast: https://talkpython.fm/episodes/show/417/test-driven-prompt-engineering-for-llms-with-promptimizeTaming AI Product Development Through Test-driven Prompt Engineering // Maxime Beauchemin // LLMs in Production Conference lightning talk: https://home.mlops.community/home/videos/taming-ai-product-development-through-test-driven-prompt-engineering --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Max on LinkedIn: https://www.linkedin.com/in/maximebeauchemin/ Timestamps: [00:00] Max introducing the Apache Superset project at Preset [01:04] Max's preferred coffee [01:16] Airflow creator [01:45] Takeaways [03:53] Please like, share, and subscribe to our MLOps channels! [04:31] Check Max's first MLOps Podcast episode [05:20] Promptimize [06:10] Interaction with API [08:27] Deterministic evaluation of SQL queries and AI [12:40] Figuring out the right edge cases [14:17] Reaction with Vector Database [15:55] Promptomize Test Suite [18:48] Promptimize vision [20:47] The open-source blood [23:04] Impact of open source [23:18] Dangers of open source [25:25] AI-Language Models Revolution [27:36] Test-driven design [29:46] Prompt tracking [33:41] Building Test Suites as Assets [36:49] Adding new prompt cases to new capabilities [39:32] Monitoring speed and cost [44:07] Creating own benchmarks [46:19] AI feature adding more value to the end users [49:39] Perceived value of the feature [50:53] LLMs costs [52:15] Specialized model versus Generalized model [56:58] Fine-tuning LLMs use cases [1:02:30] Classic Engineer's Dilemma [1:03:46] Build exciting tech that's available [1:05:02] Catastrophic forgetting [1:10:28] Promt driven development [1:13:23] Wrap up

MLOps Coffee Sessions #166 with Roy Hasson & Santona Tuli, Eliminating Garbage In/Garbage Out for Analytics and ML. // Abstract Shift left data quality ownership and observability that makes it easy for users to catch bad data at the source and stop it from entering your analytics/ML stack. // Bio Santona Tuli Santona Tuli, Ph.D. began her data journey through fundamental physics—searching through massive event data from particle collisions at CERN to detect rare particles. She’s since extended her machine learning engineering to natural language processing, before switching focus to product and data engineering for data workflow authoring frameworks. As a Python engineer, she started with the programmatic data orchestration tool, Airflow, helping improve its developer experience for data science and machine learning pipelines. Currently, at Upsolver, she leads data engineering and science, driving developer research and engagement for the declarative workflow authoring framework in SQL. Dr. Tuli is passionate about building, as well as empowering others to build, end-to-end data and ML pipelines, scalably. Roy Hasson Roy is the head of product at Upsolver helping companies deliver high-quality data to their analytics and ML tools. Previously, Roy led product management for AWS Glue and AWS Lake Formation. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links https://royondata.substack.com/ --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Roy on LinkedIn: https://www.linkedin.com/in/royhasson/ Connect with Santona on LinkedIn: https://www.linkedin.com/in/santona-tuli/ Timestamps: [00:00] Santona's and Roy's preferred coffee [01:05] Santona's and Roy's background [03:33] Takeaways [05:49] Please like, share, and subscribe to our MLOps channels! [06:42] Back story of having Santona and Roy on the podcast [09:51] Santona's story [11:37] Optimal tag teamwork [16:53] Dealing with stakeholder needs [26:25] Having mechanisms in place [27:30] Building for data Engineers vs building for data scientists [34:50] Creating solutions for users [38:55] User experience holistic point of view [41:11] Tooling sprawl is real [42:00] LLMs reliability [45:00] Things would have loved to learn five years ago [49:46] Wrap up

MLOps Coffee Sessions #165 with Sammy Sidhu, Python Power: How Daft Embeds Models and Revolutionizes Data Processing. // Abstract Sammy shares his fascinating journey in the autonomous vehicle industry, highlighting his involvement in two successful startup acquisitions by Tesla and Toyota. He emphasizes his expertise in optimizing and distilling models for efficient machine learning, which he has incorporated into his new company Eventual. The company's open-source offering, daf, focuses on tackling the challenges of unstructured and complex data. Sammy discusses the future of MLOps, machine learning, and data storage, particularly in relation to the retrieval and processing of unstructured data. The Eventual team is developing Daft, an open-source query engine that aims to provide efficient data storage solutions for unstructured data, offering features like governance, schema evolution, and time travel. The conversation sheds light on the innovative developments in the field and the potential impact on various industries. // Bio Sammy is a Deep Learning and systems veteran, holding over a dozen publications and patents in the space. Sammy graduated from the University of California, Berkeley where he did research in Deep Learning and High Performance Computing. He then joined DeepScale as the Chief Architect and led the development of perception technologies for autonomous vehicles. During this time, DeepScale grew rapidly and was subsequently acquired by Tesla in 2019. Staying in Autonomous Vehicles, Sammy joined Lyft Level 5 as a Senior Staff Software Engineer, building out core perception algorithms as well as infrastructure for machine learning and embedded systems. Level 5 was then acquired by Toyota in 2021, adopting much of his work. Sammy is now CEO and Co-Founder at Eventual Building Daft, an open-source query engine that specializes in multimodal data. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links https://sammysidhu.com/ Check out Daft, our open-source query engine for multimodal data! https://www.getdaft.io/ Here are some talks/shows we have given about it: - PyData Global (Dec 2022): Large-scale image processing: https://www.youtube.com/watch?v=ol6IQUbyeDo&ab_channel=PyData - Ray Meetup (March 2023): Distributed ML preprocessing + training on Ray https://www.youtube.com/watch?v=1MpEYlIlu7w&t=2972s&ab_channel=Anyscale - The Data Stack Show (April 2023): Self-Driving Technology and Data Infrastructure with Sammy Sidhu https://datastackshow.com/podcast/the-prql-self-driving-technology-and-data-infrastructure-with-sammy-sidhu-co-founder-and-ceo-of-eventual/ Chain of Thought for LLMs: https://cobusgreyling.medium.com/chain-of-thought-prompting-in-llms-1077164edf97 Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes: https://arxiv.org/abs/2305.02301 --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Sammy on LinkedIn: https://www.linkedin.com/in/sammy-sidhu/

MLOps Coffee Sessions #164 with Rob Hirschfeld, Open Source and Fast Decision Making. This episode is brought to you by. // Abstract Rob Hirschfeld, the CEO and co-founder of Rack N, discusses his extensive experience in the DevOps movement. He shares his notable achievement of coining the term "the cloud" and obtaining patents for infrastructure management and API provision. Rob highlights the stagnant progress in operations and the persistent challenges in security and access controls within the industry. The absence of standardization in areas such as Kubernetes and single sign-on complicates the development of robust solutions. To address these issues, Rob underscores the significance of open-source practices, automation, and version control in achieving operational independence and resilience in infrastructure management. // Bio Rob is the CEO and Co-founder of RackN, an Austin-based start-up that develops software to help automate data centers, which they call Digital Rebar. This platform helps connect all the different pieces and tools that people use to manage infrastructure into workflow pipelines through seamless multi-component automation across the different pieces and parts needed to bring up IT systems, platforms, and applications. Rob has a background in Scale Computing, Mechanical and Systems Engineering, and specializes in large-scale complex systems that are integrated with the physical environment. He has founded companies and been in the cloud and infrastructure space for nearly 25 years and has done everything from building the first Clouds using ESXi betas to serving four terms on the OpenStack Foundation Board. Rob was trained as an Industrial Engineer and holds degrees from Duke University and Louisiana State University. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links https://rackn.com/ https://robhirschfeld.com/about/ --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Rob on LinkedIn: https://www.linkedin.com/in/rhirschfeld/ Timestamps: [00:00] Rob's preferred coffee [00:17] Rob Hirschfeld's background [01:42] Takeaways [02:36] Please like, share, and subscribe to this channel! [03:09] Creation of Cloud [08:38] Changes in Cloud after 25 Years [10:54] Pros and cons of microservices [13:06] Secure Access Provisioning [15:46] Parallelism with ads [18:08] Redfish protocol [20:21] Impact of using open source vs using a SAS provider [26:15] Automation [32:39] Embrace Operational Flexibility [35:08] Automating infrastructure inefficiently [41:26] Legacy code and resiliency [43:39] Collection of metadata [45:50] RackN [51:23] Granular Cloud Preferences [54:35] Reframing of perceived complexity [57:32] Generative DevOps [58:50] Wrap up