
19 - Mechanistic Interpretability with Neel Nanda
AXRP - the AI X-risk Research Podcast
Induction Heads Are Different Kinds of Things?
There's this question like how like our induction heads all basically the same kind of thing or they're like lots of different types of things that are induction heads so you brought up anti-induction heads that can sometimes act as induction heads yes. Things are messy another form of messiness is that if but a bunch of heads have the same induction the attention patterns then it's like in some sense the model doesn't care about the fact that the oV matrices are like different between the two heads.
00:00
Transcript
Play full episode
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.