Catherine Olsson and Nelson Elhage: Anthropic, Understanding Transformers
The Gradient: Perspectives on AI
00:00
The OV Matrix and the Path Expansion Frame
QK, V and O piece are computed for efficiency reasons on the GPU somewhat separately. But if we look mathematically, in fact, the QK matrices could be viewed as doing one coherent operation. So that sort of weighted sum gets applied to the values, which then are transformed through an output matrix and then written back onto the residual stream. And so again, this sort of path expansion frame is a mathematically inspired frame.
Transcript
Play full episode
