Anton Teaches Packy AI | E1
"Age of Miracles"
Is Attention a Multi-Head Mechanism?
So first we have this thing called a multi-head attention mechanism. Attention weights the input appropriately in order to make a weighted prediction instead of a flat equally distributed prediction. Now, then it has this other thing where it says add and norm. So what that allows you to do here is, okay, we're getting like these values out of the attention heads, but we also want to keep what we had originally,. so that we can keep propagating that information as well.
00:00
Transcript
Play full episode
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
