AXRP - the AI X-risk Research Podcast cover image

19 - Mechanistic Interpretability with Neel Nanda

AXRP - the AI X-risk Research Podcast

CHAPTER

How Many Induction Heads Do You Have?

induction heads seem sudden not gradual and that strikes me as kind of weird like why do you think they appear suddenly and not just gradually throughout training or more this is a great question one thing to point out is that there just hasn't been that much investigation of different circuits and their training dynamics. If I want tons of induction heads what architecture should I use let's see gpt near seems to have a lot oh gpt2 excel have tons go use the big gpt2 architecture that's my advice okay yeahMy my next like big question about the big picture of the paper is like yeah the argument is that like induction heads there's just like this particular phase during training where like they start occurring

00:00
Transcript
Play full episode

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
App store bannerPlay store banner