AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
2min chapter
Scaling LLMs and Accelerating Adoption with Aidan Gomez at Cohere
Gradient Dissent: Conversations on AI
Transformers: A Multi-Layer Perception Work
I think Transformers are almost MLPs. So they do look like just a bunch of matmalls. They add in like one more access to do matmalls across. And that's the length of your sequence. But I think it's basically trying to cut as close to a massive MLP as you possibly can, because those do saturate compute the best. What you want to avoid are tons of little ops like softmaxes and little activation functions, dropout layers,. All of these little things which break up those big matmall.
00:00
Transcript
Episode notes
Get the Snipd
podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any
moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share
& Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode