
19 - Mechanistic Interpretability with Neel Nanda
AXRP - the AI X-risk Research Podcast
The Third Reason Why You Shouldn't Expect Face Transitions With Adam Based Optimizers
All circuits that form in a model should form as face transitions. This hypothesis predicts that if you look at the whether that circuit exists, or like the components of that circuit these form in a shift where like you go from does not have to has pretty rapidly. And there's this kind of interesting thing where each of these is only useful each part is going to use for the other parts of their formula. My hypothesis is there some kind of lottery ticket style thing going on here. The further the progress gets the stronger the gradient signal and we'll build some toy models of this.
00:00
Transcript
Play full episode
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.