AXRP - the AI X-risk Research Podcast cover image

19 - Mechanistic Interpretability with Neel Nanda

AXRP - the AI X-risk Research Podcast

CHAPTER

Induction Heads Are Different Kinds of Things?

There's this question like how like our induction heads all basically the same kind of thing or they're like lots of different types of things that are induction heads so you brought up anti-induction heads that can sometimes act as induction heads yes. Things are messy another form of messiness is that if but a bunch of heads have the same induction the attention patterns then it's like in some sense the model doesn't care about the fact that the oV matrices are like different between the two heads.

00:00
Transcript
Play full episode

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
App store bannerPlay store banner