Ashish Vaswani and Jakob Uszkoreit, co-authors of the "Attention Is All You Need" paper, discuss the motivation behind replacing RNNs and CNNs with a self-attention mechanism in the Transformer model. They delve into topics such as the positional encoding mechanism, multi-headed attention, replacing encoders in other models, and what self-attention actually learns. They highlight how lower layers learn n-grams and higher layers learn coreference, showcasing the power of the self-attention mechanism.
