The Nonlinear Library EA - Clarifying METR's Auditing Role [linkpost] by ChanaMessinger
Jun 5, 2024
01:25
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Clarifying METR's Auditing Role [linkpost], published by ChanaMessinger on June 5, 2024 on The Effective Altruism Forum.
METR clarifying what auditing it has or hasn't done, and what its plans are for the future, seems very important for understanding the landscape.
Excerpt:
METR has not intended to claim to have audited anything, or to be providing meaningful oversight or accountability, but there has been some confusion about whether METR is an auditor or planning to be one.
To clarify this point:
1. METR's top priority is to develop the science of evaluations, and we don't need to be auditors in order to succeed at this.
We aim to build evaluation protocols that can be used by evaluators/auditors regardless of whether that is the government, an internal lab team, another third party, or a team at METR.
2. We should not be considered to have 'audited' GPT-4 or Claude.
Those were informal pilots of what an audit might involve, or research collaborations - not providing meaningful oversight. For example, it was all under NDA - we didn't have any right or responsibility to disclose our findings to anyone outside the labs - and there wasn't any formal expectation it would inform deployment decisions. We also didn't have the access necessary to perform a proper evaluation.
Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
METR clarifying what auditing it has or hasn't done, and what its plans are for the future, seems very important for understanding the landscape.
Excerpt:
METR has not intended to claim to have audited anything, or to be providing meaningful oversight or accountability, but there has been some confusion about whether METR is an auditor or planning to be one.
To clarify this point:
1. METR's top priority is to develop the science of evaluations, and we don't need to be auditors in order to succeed at this.
We aim to build evaluation protocols that can be used by evaluators/auditors regardless of whether that is the government, an internal lab team, another third party, or a team at METR.
2. We should not be considered to have 'audited' GPT-4 or Claude.
Those were informal pilots of what an audit might involve, or research collaborations - not providing meaningful oversight. For example, it was all under NDA - we didn't have any right or responsibility to disclose our findings to anyone outside the labs - and there wasn't any formal expectation it would inform deployment decisions. We also didn't have the access necessary to perform a proper evaluation.
Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org