Vanishing Gradients

While many people talk about “agents,” Shreya Shankar (UC Berkeley) has been building the systems that make them reliable. In this episode, she shares how AI agents and LLM judges can be used to process millions of documents accurately and cheaply. Drawing from work on projects ranging from databases of police misconduct reports to large-scale customer transcripts, Shreya explains the frameworks, error analysis, and guardrails needed to turn flaky LLM outputs into trustworthy pipelines. We talk through: Treating LLM workflows as ETL pipelines for unstructured text Error analysis: why you need humans reviewing the first 50–100 traces Guardrails like retries, validators, and “gleaning” How LLM judges work — rubrics, pairwise comparisons, and cost trade-offs Cheap vs. expensive models: when to swap for savings Where agents fit in (and where they don’t) If you’ve ever wondered how to move beyond unreliable demos, this episode shows how to scale LLMs to millions of documents — without breaking the bank. LINKS Shreya's website DocETL, A system for LLM-powered data processing Upcoming Events on Luma Watch the podcast video on YouTube Shreya's AI evals course, which she teaches with Hamel "Evals" Husain 🎓 Learn more: Hugo's course: Building LLM Applications for Data Scientists and Software Engineers — https://maven.com/s/course/d56067f338 ($600 off early bird discount for November cohort availiable until August 31)

Ravin Kumar, a researcher at Google DeepMind, dives into the newly launched Gemma 270M, the smallest member of the Gemma 3 family of AI models. He explains its efficiency and speed, perfect for on-device use cases where privacy and latency are crucial. Kumar discusses the strategic advantages of smaller models for fine-tuning and targeted tasks, emphasizing their potential to drive broader AI adoption. Listeners will learn how to leverage 270M for specific applications and compare it with larger models in diverse scenarios.

Join Eric Ma, who heads research data science at Moderna, as he discusses the wild world of AI systems over breakfast at SciPy. He reveals why 'perfect' testing can lead you astray and introduces three key personas in AI development, each with unique blind spots. Discover how curiosity can elevate builders from good to great, and learn about maintaining observability in both development and production. Eric also shares insights on fostering experimentation in large organizations, embracing the chaos that comes with creating thriving AI products.

Zach Mueller, who leads Accelerate at Hugging Face, shares his expertise on scaling AI from cozy Colab environments to powerful clusters. He explains how to get started with just a couple of GPUs, debunks myths about performance bottlenecks, and discusses practical strategies for training on a budget. Zach emphasizes the importance of understanding distributed systems for any ML engineer and underscores how these skills can make a significant impact on their career. Tune in for actionable insights and demystifying tips!

Samuel Colvin, the mastermind behind Pydantic and founder of Logfire, discusses the often-overlooked challenges in AI reliability. He emphasizes how durability is key, not just flashy demos, and reveals that tiny feedback loops can significantly enhance performance insights. Colvin introduces innovative concepts like prompt self-repair systems and drift alarms, which can catch shifts before they become problems. He advocates for business-driven metrics that ensure features align with real goals, making AI not just functional but dependable in real-world applications.

Eric Ma, who leads data science research at Moderna, dives into the challenges of aligning retrieval with user intent in LLM-powered systems. He argues that most features fail not at the model level but with context. Eric reveals how a simple YAML-based approach can outperform complex pipelines and discusses the pitfalls of vague user queries. He also emphasizes the importance of evolving retrieval workflows to meet user needs and when it's sufficient to rely on intuition versus formal evaluation in refining these systems.

In this discussion, Philip Carter, Product Management Director at Salesforce and former Principal PM at Honeycomb, shares insights on creating LLM-powered features. He explains the nuances of integrating real production data with these systems. Carter dives into the challenges of tool use, prompt templates, and flaky model behavior. He also discusses the development of the innovative MCP server that enhances observability in AI systems, emphasizing its role in improving user experience and navigating the pitfalls of SaaS product development.

Hamel Husain, an AI specialist with experience at Airbnb, GitHub, and DataRobot, discusses improving AI products through effective evaluation. He highlights the importance of error analysis and systematic iteration in development. The conversation reveals common pitfalls in debugging AI systems, stressing the collaboration between engineers and domain experts to drive progress. Hamel also emphasizes that evaluation should be a comprehensive process, balancing immediate fixes with strategic assessment. This dialogue is a must-hear for anyone grappling with AI system enhancements.

Akshay Agrawal, founder of Marimo and former Google Brain researcher, discusses the critical challenges faced in AI at scale. He emphasizes the need for robust infrastructure over just improved models. The conversation covers the importance of reproducibility and the shortcomings of traditional tools. Akshay introduces Marimo's innovative design that addresses modular AI applications and the difficulties in debugging large language models. Live demos illustrate Marimo's capabilities in data extraction and agentic workflows, merging technical insights with cultural reflections in data science.

If we want to make progress toward AGI, we need a clear definition of intelligence—and a way to measure it. In this episode, Hugo talks with Greg Kamradt, President of the ARC Prize Foundation, about ARC-AGI: a benchmark built on Francois Chollet’s definition of intelligence as “the efficiency at which you learn new things.” Unlike most evals that focus on memorization or task completion, ARC is designed to measure generalization—and expose where today’s top models fall short. They discuss: 🧠 Why we still lack a shared definition of intelligence 🧪 How ARC tasks force models to learn novel skills at test time 📉 Why GPT-4-class models still underperform on ARC 🔎 The limits of traditional benchmarks like MMLU and Big-Bench ⚙️ What the OpenAI O₃ results reveal—and what they don’t 💡 Why generalization and efficiency, not raw capability, are key to AGI Greg also shares what he’s seeing in the wild: how startups and independent researchers are using ARC as a North Star, how benchmarks shape the frontier, and why the ARC team believes we’ll know we’ve reached AGI when humans can no longer write tasks that models can’t solve. This conversation is about evaluation—not hype. If you care about where AI is really headed, this one’s worth your time. LINKS ARC Prize -- What is ARC-AGI? On the Measure of Intelligence by François Chollet Greg Kamradt on Twitter Hugo's High Signal Podcast with Fei-Fei Li Vanishing Gradients YouTube Channel Upcoming Events on Luma Hugo's recent newsletter about upcoming events and more! Watch the podcast here on YouTube! 🎓 Want to go deeper? Check out Hugo's course: Building LLM Applications for Data Scientists and Software Engineers. Learn how to design, test, and deploy production-grade LLM systems — with observability, feedback loops, and structure built in. This isn’t about vibes or fragile agents. It’s about making LLMs reliable, testable, and actually useful. Includes over $800 in compute credits and guest lectures from experts at DeepMind, Moderna, and more. Cohort starts July 8 — Use this link for a 10% discount