Generate a Sequence Using the Attention Mechanism

When generating a sequence, the idea is that you'll have something where essentially the thing you feed into the neural net is something that is going to be. And what you're doing is you're looking back in the previous things that are already in the sequence and you're saying, okay, which numbered things in that sequence should I look at to feed it into my neural net that works out what the next thing should be. So one of the things you try to do is to learn sort of which word is worth looking at in these in in the kind of preceding part of the text.