AXRP - the AI X-risk Research Podcast cover image

19 - Mechanistic Interpretability with Neel Nanda

AXRP - the AI X-risk Research Podcast

CHAPTER

The Third Reason Why You Shouldn't Expect Face Transitions With Adam Based Optimizers

All circuits that form in a model should form as face transitions. This hypothesis predicts that if you look at the whether that circuit exists, or like the components of that circuit these form in a shift where like you go from does not have to has pretty rapidly. And there's this kind of interesting thing where each of these is only useful each part is going to use for the other parts of their formula. My hypothesis is there some kind of lottery ticket style thing going on here. The further the progress gets the stronger the gradient signal and we'll build some toy models of this.

00:00
Transcript
Play full episode

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
App store bannerPlay store banner