AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
3min chapter
Hungry Hungry Hippos - H3
Deep Papers
Are Attention Layers Doing Great Things?
Tree, Tree is mentioning or referring to a couple works that we actually don't have out yet, but we'll be releasing probably in the next couple of weeks. We did some deep dives into exactly kind of what these remaining attention layers are doing. And we found that maybe it's possible to, to kind of remove them. One super simple idea that that we had was just, one of our landmates named Michael Polly came up with and he thought, okay, I'll just add more. So, so we're very excited about that. But if you only have two or three attention layers and you scale up really long, you only have to pay the cost of long context for one
00:00
Transcript
Episode notes
Get the Snipd
podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any
moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share
& Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode