The Scaling Laws and Scaling Loss of Language Models

In general, the scaling laws results are very impressive and I put a lot of weight on them continuing to hold. But that being said, yeah, what we're specifically looking at is kind of edge cases and the potential for some failure modes to kind of slip through. And maybe the larger models are more prone to this is one hypothesis. There's another kind of class of failures related to the inductive biases of the models. When you feed them some text, they're likely to kind of repeat it even when it's not necessarily appropriate or get caught in loops.