Continuous Improvement in LLM Deployment

Hugo speaks with Shreya Shankar, a researcher at UC Berkeley focusing on data management systems with a human-centered approach. Shreya's work is at the cutting edge of human-computer interaction (HCI) and AI, particularly in the realm of large language models (LLMs). Her impressive background includes being the first ML engineer at Viaduct, doing research engineering at Google Brain, and software engineering at Facebook.
In this episode, we dive deep into the world of LLMs and the critical challenges of building reliable AI pipelines. We'll explore:
The fascinating journey from classic machine learning to the current LLM revolution
Why Shreya believes most ML problems are actually data management issues
The concept of "data flywheels" for LLM applications and how to implement them
The intriguing world of evaluating AI systems - who validates the validators?
Shreya's work on SPADE and EvalGen, innovative tools for synthesizing data quality assertions and aligning LLM evaluations with human preferences
The importance of human-in-the-loop processes in AI development
The future of low-code and no-code tools in the AI landscape
We'll also touch on the potential pitfalls of over-relying on LLMs, the concept of "Habsburg AI," and how to avoid disappearing up our own proverbial arseholes in the world of recursive AI processes.
Whether you're a seasoned AI practitioner, a curious data scientist, or someone interested in the human side of AI development, this conversation offers valuable insights into building more robust, reliable, and human-centered AI systems.
LINKS
The livestream on YouTube (https://youtube.com/live/hKV6xSJZkB0?feature=share)
Shreya's website (https://www.sh-reya.com/)
Shreya on Twitter (https://x.com/sh_reya)
Data Flywheels for LLM Applications (https://www.sh-reya.com/blog/ai-engineering-flywheel/)
SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines (https://arxiv.org/abs/2401.03038)
What We’ve Learned From A Year of Building with LLMs (https://applied-llms.org/)
Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences (https://arxiv.org/abs/2404.12272)
Operationalizing Machine Learning: An Interview Study (https://arxiv.org/abs/2209.09125)
Vanishing Gradients on Twitter (https://twitter.com/vanishingdata)
Hugo on Twitter (https://twitter.com/hugobowne)
In the podcast, Hugo also mentioned that this was the 5th time he and Shreya chatted publicly. which is wild!
If you want to dive deep into Shreya's work and related topics through their chats, you can check them all out here:
Outerbounds' Fireside Chat: Operationalizing ML -- Patterns and Pain Points from MLOps Practitioners (https://www.youtube.com/watch?v=7zB6ESFto_U)
The Past, Present, and Future of Generative AI (https://youtu.be/q0A9CdGWXqc?si=XmaUnQmZiXL2eagS)
LLMs, OpenAI Dev Day, and the Existential Crisis for Machine Learning Engineering (https://www.youtube.com/live/MTJHvgJtynU?si=Ncjqn5YuFBemvOJ0)
Lessons from a Year of Building with LLMs (https://youtube.com/live/c0gcsprsFig?feature=share)
Check out and subcribe to our lu.ma calendar (https://lu.ma/calendar/cal-8ImWFDQ3IEIxNWk) for upcoming livestreams!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit hugobowne.substack.com