Inverse Scaling Tasks - What You've Learned So Far

In inverse scaling tasks we find that the performance on the smaller models starts out about random. And then the inverse scaling starts when the model kind of quote unquote thinks that it understands how to do the task but really is getting it wrong from our perspective, and so becomes more confident on the wrong answer. One way that this can happen is that you have what's quite quite a hard task and within that is an easier task. They may get big enough to be able to solve the harder task as well and then their performance will start to improve again. But I think there's some hints that as models get bigger they'll start to,. for example, in my example with the harder task and