Revolutionizing Language Modeling with Excel STM Models

The podcast episode delves into the revolutionary advancements in language modeling with the introduction of Excel STM models. These models aim to scale LSTM technology to billions of parameters by introducing exponential gating and a matrix memory combined with a covariance update rule. Through comparisons with CR transformers and state space models like GPT and Mamba, the Excel STM models demonstrate superior performance in word prediction and scalability with model sizes ranging from 125 million to 1.3 billion parameters. Despite achieving better results, the Excel STM models require less computing power and offer faster inference, signaling a significant leap in language modeling technology.