Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

This paper presents a novel approach to scaling Transformer-based large language models to handle infinitely long inputs with bounded memory and computation.

The Infini-attention mechanism incorporates compressive memory into the vanilla attention mechanism, enabling efficient processing of long contexts.

The approach is demonstrated on various language modeling benchmarks.