“The corrigibility basin of attraction is a misleading gloss” by Jeremy Gillen

Dec 6, 2025

Dive into the complexities of AGI as Jeremy Gillen challenges the idea of corrigibility and its role in designing safer AI. He introduces the basin-of-attraction concept, arguing that merely approximating human values may fall short. With engaging analogies like a cargo ship, he highlights potential pitfalls in iterative engineering. Jeremy critiques common assumptions and suggests the need for clearer theoretical frameworks. Get ready for an insightful discussion on the balance between empirical fixes and robust understanding in AGI development!

Ask episode

AI Snips

Chapters

Transcript

Episode notes

INSIGHT

Basin-of-Attraction Intuition

The basin-of-attraction idea says corrigible AIs will help make successors corrigible, creating an attractor for safe iteration.
This motivates prosaic alignment by enlarging the acceptable initial target and enabling iterative improvement.

INSIGHT

Corrigibility Requires Self-Doubt

Eliezer's original corrigibility demands the agent understand its own fallibility and avoid becoming overconfident.
This is hard because systems that fully trust their reasoning tend to abandon rule-like fail-safes they were given.

INSIGHT

Different Corrigibility Flavors

Paul's and Max's versions implement corrigibility differently, often via act-based objectives or principles to empower.
These versions emphasize iterative human-AI collaboration but may lose the explicit self-flaw-awareness of MIRI corrigibility.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

The idea of a “basin of attraction around corrigibility” motivates much of prosaic alignment research. Essentially this is an abstract way of thinking about the process of iteration on AGI designs. Engineers test to find problems, then understand the problems, then design fixes. The reason we need corrigibility for this is that a non-corrigible agent generally has incentives to interfere with this process. The concept was introduced by Paul Christiano:

... a corrigible agent prefers to build other agents that share the overseer's preferences — even if the agent doesn’t yet share the overseer's preferences perfectly. After all, even if you only approximately know the overseer's preferences, you know that the overseer would prefer the approximation get better rather than worse.
Thus an entire neighborhood of possible preferences lead the agent towards the same basin of attraction. We just have to get “close enough” that we are corrigible, we don’t need to build an agent which exactly shares humanity's values, philosophical views, or so on.
In addition to making the initial target bigger, this gives us some reason to be optimistic about the dynamics of AI systems iteratively designing new AI systems. Corrigible systems want to design more corrigible [...]

---

Outline:

(02:40) Corrigibility

(07:45) My argument: The engineering feedback loop will use up all its fuel

(08:31) There are many fields where test-driven engineering is far from sufficient

(10:41) The corresponding story for iteratively building corrigible AGI

(12:18) How do we know the distribution shift will stress parts of the AGI?

(16:13) Tying this back to the basin of attraction

(18:09) A counterargument: The prosaic case

(21:19) More specific counterarguments

(25:07) One way things could go well

(27:00) Conclusion and implications

The original text contained 16 footnotes which were omitted from this narration.

---

First published:
December 6th, 2025

Source:
https://www.lesswrong.com/posts/oLbpfPkdtcknABvvw/the-corrigibility-basin-of-attraction-is-a-misleading-gloss

---

Narrated by TYPE III AUDIO.