Catherine Olsson and Nelson Elhage: Anthropic, Understanding Transformers
The Gradient: Perspectives on AI
What Are Induction Heads?
An induction head looks for an exact match with the present token and then emitting the literal following token that followed last time, it will predict follows again this time. This is kind of a strict literal induction heads and they can often do sort of softer induction behaviors as well. For example, last time in the current context, there was a token like this one, it emitted a token. So let's submit a token like that one. These softening of the semantics also seem to show up as well.
00:00
Transcript
Play full episode
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
