How Chat GPT Works

Chat GPT works by generating a word at a time, sometimes it's a subword and you can see that it makes up words because it has tokens that are not quite a whole word. And so it can end up gluing together those two tokens to make up a word that's never been seen before. That's the distance back that it's kind of a tension system looks to make, to keep coherence in the text. If it was talking at Blue Wolves at the beginning, if you ask it to go too long, it'll be talking about orange, you know, elephants or something at the end. But that's how far the training could be made to work.