Transformer Memory as a Differentiable Search Index: memorizing thousands of random doc ids works!?
Neural Search Talks — Zeta Alpha
00:00
Using Direct Indexing to Get the First 32 Tokens of a Document
The simplest thing to do here is just to take the first, they call them L tokens. So you have a document, you say, I'll take the first 32 tokens. This is the common setting. And then you want to store these 32 tokens in the index in a way that associates them with the right doc ID. But that's, they call direct indexing. The other approach kind of feels like trying to go beyond this issue of only looking at the first 32 token. They come up with different chunks of the document and try to store them. That sounds like a pretty short context in modern transformers and stuff. Do you have any idea why they would only do that?
Transcript
Play full episode
