Mixture-of-Depths: Dynamically allocating compute in transformer-based language models

This paper presents a novel method called Mixture-of-Depths, which dynamically allocates computational resources in transformer-based language models.

By adapting the depth of the transformer layers based on input complexity, the approach improves efficiency and performance.

The authors demonstrate significant improvements in speed and accuracy across various language tasks.