Arxiv paper - From Bytes to Ideas: Language Modeling with Autoregressive U-Nets

In this episode, we discuss From Bytes to Ideas: Language Modeling with Autoregressive U-Nets by Mathurin Videau, Badr Youbi Idrissi, Alessandro Leite, Marc Schoenauer, Olivier Teytaud, David Lopez-Paz. The paper introduces an autoregressive U-Net model that dynamically learns its own token embeddings from raw bytes instead of relying on fixed tokenization schemes like BPE. This multi-scale architecture processes text from fine-grained bytes to broader semantic units, enabling predictions at varying future horizons. The approach matches strong baselines with shallow hierarchies and shows potential improvements with deeper ones, offering flexibility across languages and tasks.