"Age of Miracles" cover image

Anton Teaches Packy AI | E1

"Age of Miracles"

00:00

Is Attention a Multi-Head Mechanism?

So first we have this thing called a multi-head attention mechanism. Attention weights the input appropriately in order to make a weighted prediction instead of a flat equally distributed prediction. Now, then it has this other thing where it says add and norm. So what that allows you to do here is, okay, we're getting like these values out of the attention heads, but we also want to keep what we had originally,. so that we can keep propagating that information as well.

Transcript
Play full episode

The AI-powered Podcast Player

Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!
App store bannerPlay store banner
Get the app