AXRP - the AI X-risk Research Podcast cover image

19 - Mechanistic Interpretability with Neel Nanda

AXRP - the AI X-risk Research Podcast

CHAPTER

How to Improve the Loss of a Token

The paper does make the point that if you compare the models per token loss at different points in the context to the smaller to the smallest model it like has gotten most of the benefit. One hypothesis is that just a lot of the stuff that models do is not actually that relevant to changing the loss because there's just so many tasks they're getting increasingly niche and so a bunch of the things around like obviously a tiny model could not track long range dependencies in this way just aren't important enough to really matter, he says.

00:00
Transcript
Play full episode

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
App store bannerPlay store banner