The Unseen Benefits of Large Language Models

Amitai Etzioni: It's unclear to me why predicting the next word would give rise to linguistic representations that can actually do a very broad range of tasks. In school, I was taught, you know, don't make your models too expressive because they'll just memorize. And yet we're seeing this weird counter phenomenon with scaling where actually by adding more and more parameters, you're overfitting less, you're generalizing better. He says OpenAI has all these GPUs and like this is like problematic in a lot of ways.