Scaling Loss and the Importance of Data

The more data you have, the better your model is. But for transfer learning to other tasks, then the relationship becomes a bit more blurry. At some points, you need diminishing returns and you approach an asymptotic performance when you have infinite data if you need compute. This kind of is goodness for the previous paper, because at some point, then you don't need additional data. Well, I would actually say it's bad news, because if you want the same improvement that you had from in a given year, then you need to increase the amount of data even more next year.