AI Gateway, Orchestration Layers, and Evaluation Methods for Text-to-Sequel Models

Laurel Orr is a Principal Engineer at Numbers Station, a startup that applies Foundation Model technology to the enterprise data stack. Venky Orr is SVP of Product & Engineering.

MLOps podcast #196 with Numbers Station's Venky Ganti SVP, Product & Engineering and Principal Engineer, Laurel Orr, LLMs in Focus: From One-Size Fits All to Verticalized Solutions. // Abstract Dive into the realm of large language models (LLMs) as we explore the merits and limitations of 'one-size fits all' LLMs, and their role in data analytics. Through customer stories, we showcase real-world applications and contrast general LLMs with verticalized, enterprise-centric models. We address the significance of ownership structures, with a focus on open-source vs proprietary impacts on transparency and trustworthiness. Delving into the NSQL foundation models, we emphasize the importance of diverse, quality training data, especially with enterprise challenges. Lastly, we speculate on the future of LLMs, highlighting hosting solutions and the evolution towards specialized challenges. // Bio Laurel Orr Laurel Orr is a Principal Engineer at Numbers Station, a startup that applies Foundation Model technology to the enterprise data stack. Her research interests include how to use FMs to solve classically hard data-wrangling tasks and how to put FM technology into deployment. Before Numbers Station, Laurel was a postdoc at Stanford advised by Chris Re as part of the Hazy Research Labs working in the intersection of AI and data management. She graduated with a PhD in database systems from the University of Washington. Venky Orr Venky brings over two decades of experience in software engineering and technical leadership to Numbers Station as SVP of Product & Engineering. Most recently, he served as General Manager leading several initiatives on query understanding and commerce in the ads product area at Google. Before that, he was CEO and co-founder of Mesh Dynamics, the API test automation company, which was acquired by Google in 2021. Prior to Mesh Dynamics, Venky was CTO and co-founder of Alation, the enterprise data catalog company, where he led technology and helped create the new data catalog product category. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links Website: https://www.numbersstation.ai/ ⁠ --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Laurel on LinkedIn: https://www.linkedin.com/in/laurel-orr/ Connect with Venky on LinkedIn: https://www.linkedin.com/in/venky-ganti-2679a2/ Timestamps: [00:00] Venky's and Laurel's preferred coffee [00:36] Takeaways [03:15] Please like, share, and subscribe to our MLOps channels! [04:38] Venky's background [07:47] Laurel's at background [09:38] Data wrangling [13:45] Sequel query [19:25] One size-fits-all LLMs vs Verticalized and Specific LLMs [23:42] Model Choice Trade-offs [30:18] NSQL Foundational Models [37:26] LLM Trends in 12 Months [40:09] Data recipes being democratized [45:16] Claude and 100,000 Context [48:02] Exploring Varieties of LLMs [50:02] AI Gateway [51:07] Text-to-SQL Model Evaluation [54:00] Wrap up