Catherine Olsson and Nelson Elhage: Anthropic, Understanding Transformers
The Gradient: Perspectives on AI
00:00
The Intact Context Learning Score
In context learning is this ability to make use of information in its input to refine, improve the output later as it continues outputting text. The basic way that transformers are trained is that at every position, they look at all of the preceding text and then they try to predict the next token. And so you can measure the accuracy of that prediction in a number of ways. Each of those how good the prediction is, you subtract those two losses and we define that as the in context learning score.
Transcript
Play full episode
