
19 - Mechanistic Interpretability with Neel Nanda
AXRP - the AI X-risk Research Podcast
How to Improve the Loss of a Token
The paper does make the point that if you compare the models per token loss at different points in the context to the smaller to the smallest model it like has gotten most of the benefit. One hypothesis is that just a lot of the stuff that models do is not actually that relevant to changing the loss because there's just so many tasks they're getting increasingly niche and so a bunch of the things around like obviously a tiny model could not track long range dependencies in this way just aren't important enough to really matter, he says.
00:00
Transcript
Play full episode
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.