AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Scaling Plateaus Before We Reach Human Level Intelligence
I think it's very unlikely that the scaling laws will just stop. I wouldn't bet on either of those things happening, but they could. If somehow, and I don't know how we would know this, it kind of wasn't about the architecture, and we just hit a wall, I'd be very surprised by that. And there are some things that are just very hard to represent when you don't have this ability to attend far in the past that transformers have. But if you told me your model was bigger than yep, and every work didn't work out for a number of reasons, I'm not sure what my answer is.