The Nonlinear Library LW - Maybe Anthropic's Long-Term Benefit Trust is powerless by Zach Stein-Perlman
May 27, 2024
04:43
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Maybe Anthropic's Long-Term Benefit Trust is powerless, published by Zach Stein-Perlman on May 27, 2024 on LessWrong.
Crossposted from AI Lab Watch. Subscribe on Substack.
Introduction
Anthropic has an unconventional governance mechanism: an independent "Long-Term Benefit Trust" elects some of its board. Anthropic sometimes emphasizes that the Trust is an experiment, but mostly points to it to argue that Anthropic will be able to promote safety and benefit-sharing over profit.[1]
But the Trust's details have not been published and some information Anthropic has shared is concerning. In particular, Anthropic's stockholders can apparently overrule, modify, or abrogate the Trust, and the details are unclear.
Anthropic has not publicly demonstrated that the Trust would be able to actually do anything that stockholders don't like.
The facts
There are three sources of public information on the Trust:
The Long-Term Benefit Trust (Anthropic 2023)
Anthropic Long-Term Benefit Trust (Morley et al. 2023)
The $1 billion gamble to ensure AI doesn't destroy humanity (Vox: Matthews 2023)
They say there's a new class of stock, held by the Trust/Trustees. This stock allows the Trust to elect some board members and will allow them to elect a majority of the board by 2027.
But:
1. Morley et al.: "the Trust Agreement also authorizes the Trust to be enforced by the company and by groups of the company's stockholders who have held a sufficient percentage of the company's equity for a sufficient period of time," rather than the Trustees.
1. I don't know what this means.
2. Morley et al.: the Trust and its powers can be amended "by a supermajority of stockholders. . . . [This] operates as a kind of failsafe against the actions of the Voting Trustees and safeguards the interests of stockholders." Anthropic: "the Trust and its powers [can be changed] without the consent of the Trustees if sufficiently large supermajorities of the stockholders agree."
1. It's impossible to assess this "failsafe" without knowing the thresholds for these "supermajorities." Also, a small number of investors - currently, perhaps Amazon and Google - may control a large fraction of shares. It may be easy for profit-motivated investors to reach a supermajority.
3. Maybe there are other issues with the Trust Agreement - we can't see it and so can't know.
4. Vox: the Trust "will elect a fifth member of the board this fall," viz. Fall 2023.
1. Anthropic has not said whether that happened nor who is on the board these days (nor who is on the Trust these days).
Conclusion
Public information is consistent with the Trust being quite subordinate to stockholders, likely to lose their powers if they do anything stockholders dislike. (Even if stockholders' formal powers over the Trust are never used, that threat could prevent the Trust from acting contrary to the stockholders' interests.)
Anthropic knows this and has decided not to share the information that the public needs to evaluate the Trust. This suggests that Anthropic benefits from ambiguity because the details would be seen as bad. I basically fail to imagine a scenario where publishing the Trust Agreement is very costly to Anthropic - especially just sharing certain details (like sharing percentages rather than saying "a supermajority") - except that the details are weak and would make Anthropic look bad.[2]
Maybe it would suffice to let an auditor see the Trust Agreement and publish their impression of it. But I don't see why Anthropic won't publish it.
Maybe the Trust gives Anthropic strong independent accountability - or rather, maybe it will by default after (unspecified) time- and funding-based milestones. But only if Anthropic's board and stockholders have substantially less power over it than they might - or if they will exercise great restraint in using their p...
Crossposted from AI Lab Watch. Subscribe on Substack.
Introduction
Anthropic has an unconventional governance mechanism: an independent "Long-Term Benefit Trust" elects some of its board. Anthropic sometimes emphasizes that the Trust is an experiment, but mostly points to it to argue that Anthropic will be able to promote safety and benefit-sharing over profit.[1]
But the Trust's details have not been published and some information Anthropic has shared is concerning. In particular, Anthropic's stockholders can apparently overrule, modify, or abrogate the Trust, and the details are unclear.
Anthropic has not publicly demonstrated that the Trust would be able to actually do anything that stockholders don't like.
The facts
There are three sources of public information on the Trust:
The Long-Term Benefit Trust (Anthropic 2023)
Anthropic Long-Term Benefit Trust (Morley et al. 2023)
The $1 billion gamble to ensure AI doesn't destroy humanity (Vox: Matthews 2023)
They say there's a new class of stock, held by the Trust/Trustees. This stock allows the Trust to elect some board members and will allow them to elect a majority of the board by 2027.
But:
1. Morley et al.: "the Trust Agreement also authorizes the Trust to be enforced by the company and by groups of the company's stockholders who have held a sufficient percentage of the company's equity for a sufficient period of time," rather than the Trustees.
1. I don't know what this means.
2. Morley et al.: the Trust and its powers can be amended "by a supermajority of stockholders. . . . [This] operates as a kind of failsafe against the actions of the Voting Trustees and safeguards the interests of stockholders." Anthropic: "the Trust and its powers [can be changed] without the consent of the Trustees if sufficiently large supermajorities of the stockholders agree."
1. It's impossible to assess this "failsafe" without knowing the thresholds for these "supermajorities." Also, a small number of investors - currently, perhaps Amazon and Google - may control a large fraction of shares. It may be easy for profit-motivated investors to reach a supermajority.
3. Maybe there are other issues with the Trust Agreement - we can't see it and so can't know.
4. Vox: the Trust "will elect a fifth member of the board this fall," viz. Fall 2023.
1. Anthropic has not said whether that happened nor who is on the board these days (nor who is on the Trust these days).
Conclusion
Public information is consistent with the Trust being quite subordinate to stockholders, likely to lose their powers if they do anything stockholders dislike. (Even if stockholders' formal powers over the Trust are never used, that threat could prevent the Trust from acting contrary to the stockholders' interests.)
Anthropic knows this and has decided not to share the information that the public needs to evaluate the Trust. This suggests that Anthropic benefits from ambiguity because the details would be seen as bad. I basically fail to imagine a scenario where publishing the Trust Agreement is very costly to Anthropic - especially just sharing certain details (like sharing percentages rather than saying "a supermajority") - except that the details are weak and would make Anthropic look bad.[2]
Maybe it would suffice to let an auditor see the Trust Agreement and publish their impression of it. But I don't see why Anthropic won't publish it.
Maybe the Trust gives Anthropic strong independent accountability - or rather, maybe it will by default after (unspecified) time- and funding-based milestones. But only if Anthropic's board and stockholders have substantially less power over it than they might - or if they will exercise great restraint in using their p...