The Limits of Intelligence

OpenAI and some others did is to see what happens if you train a very large model on the vast amount of data available on the internet. It turns out it can pick up on headers that we thought would be impossible for language models to pick up on. And although the learning objective boils down to something as simple as predicting which word comes next, the representation that is learned by these models are so powerful they can then be used for downstream NLP applications.