Context rot experiments and implications

Jeff walks through experiments showing performance degradation as context increases and its engineering impact.

Play episode from 32:45

chevron_right

Transcript

chevron_right

Transcript

Episode notes

We’re really moving from a world where humans are authoring search queries and humans are executing those queries and humans are digesting the results to a world where AI is doing that for us.

Jeff Huber, CEO and co-founder of Chroma, joins Hugo to talk about how agentic search and retrieval are changing the very nature of search and software for builders and users alike.

We Discuss:

* “Context engineering”, the strategic design and engineering of what context gets fed to the LLM (data, tools, memory, and more), which is now essential for building reliable, agentic AI systems;

* Why simply stuffing large context windows is no longer feasible due to “context rot” as AI applications become more goal-oriented and capable of multi-step tasks

* A framework for precisely curating and providing only the most relevant, high-precision information to ensure accurate and dependable AI systems;

* The “agent harness”, the collection of tools and capabilities an agent can access, and how to construct these advanced systems;

* Emerging best practices for builders, including hybrid search as a robust default, creating “golden datasets” for evaluation, and leveraging sub-agents to break down complex tasks

* The major unsolved challenge of agent evaluation, emphasizing a shift towards iterative, data-centric approaches.

You can also find the full episode on Spotify, Apple Podcasts, and YouTube.

You can also interact directly with the transcript here in NotebookLM: If you do so, let us know anything you find in the comments!

👉 Want to learn more about Building AI-Powered Software? Check out our Building AI Applications course. It’s a live cohort with hands on exercises and office hours. Our final cohort is in Q1, 2206. Here is a 35% discount code for readers. 👈

Oh! One more thing: we’ve just announced a Vanishing Gradients livestream for January 21 that you may dig:

* A Builder’s Guide to Agentic Search & Retrieval with Doug Turnbull and John Berryman (register to join live or get the recording afterwards.

Show notes

* Jeff Huber on Twitter

* Jeff Huber on LinkedIn

* Try Chroma!

* Context Rot: How Increasing Input Tokens Impacts LLM Performance by The Chroma Team

* AI Agent Harness, 3 Principles for Context Engineering, and the Bitter Lesson Revisited

* From Context Engineering to AI Agent Harnesses: The New Software Discipline

* Generative Benchmarking by The Chroma Team

* Effective context engineering for AI agents by The Anthropic Team

* Making Sense of Millions of Conversations for AI Agents by Ivan Leo (Manus) and Hugo