AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
The Evolution of Attention in Language Models
Last year, I spent a bunch of time trying to make attention as fast as possible. So we wanted to understand from an academic perspective, when or why do we need attention? Can we have other alternatives that maybe scale better in terms of sequence length? Because the longer context length has been a big problem for attention for a long time. And since then, Dan Fue and Michael Paulies have been pushing on that direction. Some things like long convolution, hyena, hierarchy,. They've seen pretty positive results. But it's definitely more of a contrarian take.