Can We Use State-Based Models in Language Modeling?

Language modeling and language models these days have kind of one major bottleneck that we wanted to address, which is they really struggle to capture long context. And our lab has been working on these primitives called state-based models for a couple years now. So with a new model, we had to put in some work to make sure that it runs at least as fast as attention on modern hardware.

Transcript

Play full episode

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.