AXRP - the AI X-risk Research Podcast cover image

19 - Mechanistic Interpretability with Neel Nanda

AXRP - the AI X-risk Research Podcast

CHAPTER

Is There More Than One Induction Head?

An induction head is something which does that on a like random string of tokens. It's not clear why you need more than one of these because it feels like one task. We force the model to squash all of the information it moves into this 64-dimensional head dimension but if you've got two heads that have exactly the same attention pattern then it de facto is compressing the residual stream into a 128-dimensional thing right? This is just higher fidelity and like maybe that's useful for doing strict induction behavior  to predict what will come next.

00:00
Transcript
Play full episode

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
App store bannerPlay store banner