MLOps.community Context Engineering, Context Rot, & Agentic Search with the CEO of Chroma, Jeff Huber
187 snips
Nov 21, 2025 
Jeff Huber, CEO of Chroma, reveals the challenges of 'context rot,' where AI memory decays, impacting performance. He discusses why traditional benchmarks can mislead developers and explains how Chroma's two-stage retrieval optimizes both recall and precision. The conversation dives into the evolution of search technologies, pitfalls of single embeddings, and the intricacies of personalization in semantic search. Huber emphasizes the need for cleaner, engineered solutions in AI that reduce dependency on fragile systems.
AI Snips
Chapters
Transcript
Episode notes
Context Windows Are Spiky
- Large context windows degrade model attention and reasoning despite strong needle-in-haystack performance.
- Jeff Huber shows multi-model tests proving behavior changes with longer contexts and diverse tasks.
AI Search Isn't Classic Search
- AI search workloads differ from classic search because many teams each host multiple indexes.
- LLMs query and digest far more results than humans did, changing retrieval design needs.
Two-Stage Retrieval Pipeline
- First maximize recall to gather all potentially relevant information before narrowing results.
- Then apply precision to filter that pool into the concise context for final model reasoning.