Catherine Olsson and Nelson Elhage: Anthropic, Understanding Transformers
The Gradient: Perspectives on AI
The OV Matrix and the Path Expansion Frame
QK, V and O piece are computed for efficiency reasons on the GPU somewhat separately. But if we look mathematically, in fact, the QK matrices could be viewed as doing one coherent operation. So that sort of weighted sum gets applied to the values, which then are transformed through an output matrix and then written back onto the residual stream. And so again, this sort of path expansion frame is a mathematically inspired frame.
00:00
Transcript
Play full episode
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
