How Much Data Do You Need?

If you give it a big piece of a well known document, it will continue verbatim because it will have enough information that's encoded. In fact, the number of weights is very comparable to the number of tokens of training data. But if I'm pretty sure if you just take a random sentence from a random blog in the internet, I think you're right. It would be able to replicate something that was well established well. And by the way, it's not good at extrapolating.