AXRP - the AI X-risk Research Podcast cover image

19 - Mechanistic Interpretability with Neel Nanda

AXRP - the AI X-risk Research Podcast

CHAPTER

Using Induction Heads to Match Translation Heads

This scales linearly with the number of features rather than quadratically like lookup tables do hmm so this is much more efficient. The kind of induction we've been discussing so far is the thing where the features you're matching are just literally token values yep but you can totally imagine this being done more abstractly for example we found this translation head which could attend from a word in a sentence in English to this relevant to the thing immediately after the relevant word in the French sentence. It's easy to match them note that I'm what I'm describing is actually more like a duplicate token head whereas I did attending from like a token to copies of itself but if you have that extending it

00:00
Transcript
Play full episode

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
App store bannerPlay store banner