Analysis of Scale AI's Language Model Leaderboards and Human Evaluations

Scale AI has introduced language model leaderboards called SEAL leaderboards to evaluate AI model performance in specific domains such as coding and instruction following. They address the issue of gaming leaderboards by conducting human evaluations instead of automated ones. This approach is considered more robust since human evaluations provide a clearer picture of model performance compared to AI evaluations like GP4. The leaderboards use elo-scale rankings to pit models against each other, ensuring a relative ranking that cannot be easily manipulated. The evaluation process involves having human evaluators determine the winner after the models compete. Scale AI's effort is commendable as they bring in experts to evaluate the models, which helps in understanding how the models stack up against each other. The data shared indicates that Cloud 3 Opus performs best in math, while GPT-4 excels across various categories, albeit by a small margin in some cases. Overall, Scale AI's approach of using human evaluations in their language model leaderboards is a significant step towards addressing the challenges of determining the best models in different domains.