19 - Mechanistic Interpretability with Neel Nanda
AXRP - the AI X-risk Research Podcast
The Induction Heads in Large Models Aren't as Important as They Used to Be
Smeared key models do not have an induction bump and do not have a phase transition by the form induction heads or get way better at in context learning. One layer models are still pretty good at in contextlearning if they have smeared keys okay and are much better than if they don't. This gives like strong credence to the our mechanistic our mechanistic claim about the circuit is like pretty importantOkay though it does not prove that all induction heads in large models are because of this circuit or that this is actually the way it is used in large models just that it's important part anyway All right so like the key thing here is just like look at the lines of evidence and say do
Remember Everything You Learn from Podcasts
