The Holistic Evaluation of Language Models

Helm is a new project that's, it's really ongoing. We released the initial version about a month ago, but the idea is that for it to be continuous out of monitoring the state of language models which is moving incredibly fast. So where you can, you can kind of select what metric you want to evaluate on or metrics and then scenarios. And so as you said, Ben, we looked at 30 models across a number of different organizations, including OpenAI,. AI 21, cohere, Google, meta, Microsoft, and then some open models from big science like Yandex and Chinghua.