The State of Scaling Loss in Machine Learning

The goal for studying scaling loss generally is to better predict and better design machine learning models. We don't quite understand theoretically why they are what they are, which tell you exactly to what extent there's diminishing returns. So it's hard to tell whether this empirical phenomenon is very concrete to our current architectures or whether it will generalize if five years from now we discover something even better than the transformer.