The Nonlinear Library: LessWrong LW - Ten arguments that AI is an existential risk by KatjaGrace
Aug 13, 2024
10:43
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ten arguments that AI is an existential risk, published by KatjaGrace on August 13, 2024 on LessWrong.
This is a snapshot of a new page on the AI Impacts Wiki.
We've made a list of arguments[1] that AI poses an existential risk to humanity. We'd love to hear how you feel about them in the comments and polls.
Competent non-aligned agents
Summary:
1. Humans will build AI systems that are 'agents', i.e. they will autonomously pursue goals
2. Humans won't figure out how to make systems with goals that are compatible with human welfare and realizing human values
3. Such systems will be built or selected to be highly competent, and so gain the power to achieve their goals
4. Thus the future will be primarily controlled by AIs, who will direct it in ways that are at odds with long-run human welfare or the realization of human values
Selected counterarguments:
It is unclear that AI will tend to have goals that are bad for humans
There are many forms of power. It is unclear that a competence advantage will ultimately trump all others in time
This argument also appears to apply to human groups such as corporations, so we need an explanation of why those are not an existential risk
People who have favorably discussed[2] this argument (specific quotes here): Paul Christiano (2021), Ajeya Cotra (2023), Eliezer Yudkowsky (2024), Nick Bostrom (2014[3]).
See also: Full wiki page on the competent non-aligned agents argument
Second species argument
Summary:
1. Human dominance over other animal species is primarily due to humans having superior cognitive and coordination abilities
2. Therefore if another 'species' appears with abilities superior to those of humans, that species will become dominant over humans in the same way
3. AI will essentially be a 'species' with superior abilities to humans
4. Therefore AI will dominate humans
Selected counterarguments:
Human dominance over other species is plausibly not due to the cognitive abilities of individual humans, but rather because of human ability to communicate and store information through culture and artifacts
Intelligence in animals doesn't appear to generally relate to dominance. For instance, elephants are much more intelligent than beetles, and it is not clear that elephants have dominated beetles
Differences in capabilities don't necessarily lead to extinction. In the modern world, more powerful countries arguably control less powerful countries, but they do not wipe them out and most colonized countries have eventually gained independence
People who have favorably discussed this argument (specific quotes here): Joe Carlsmith (2024), Richard Ngo (2020), Stuart Russell (2020[4]), Nick Bostrom (2015).
See also: Full wiki page on the second species argument
Loss of control via inferiority
Summary:
1. AI systems will become much more competent than humans at decision-making
2. Thus most decisions will probably be allocated to AI systems
3. If AI systems make most decisions, humans will lose control of the future
4. If humans have no control of the future, the future will probably be bad for humans
Selected counterarguments:
Humans do not generally seem to become disempowered by possession of software that is far superior to them, even if it makes many 'decisions' in the process of carrying out their will
In the same way that humans avoid being overpowered by companies, even though companies are more competent than individual humans, humans can track AI trustworthiness and have AI systems compete for them as users. This might substantially mitigate untrustworthy AI behavior
People who have favorably discussed this argument (specific quotes here): Paul Christiano (2014), Ajeya Cotra (2023), Richard Ngo (2024).
See also: Full wiki page on loss of control via inferiority
Loss of control via speed
Summary:
1. Advances in AI will produce...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ten arguments that AI is an existential risk, published by KatjaGrace on August 13, 2024 on LessWrong.
This is a snapshot of a new page on the AI Impacts Wiki.
We've made a list of arguments[1] that AI poses an existential risk to humanity. We'd love to hear how you feel about them in the comments and polls.
Competent non-aligned agents
Summary:
1. Humans will build AI systems that are 'agents', i.e. they will autonomously pursue goals
2. Humans won't figure out how to make systems with goals that are compatible with human welfare and realizing human values
3. Such systems will be built or selected to be highly competent, and so gain the power to achieve their goals
4. Thus the future will be primarily controlled by AIs, who will direct it in ways that are at odds with long-run human welfare or the realization of human values
Selected counterarguments:
It is unclear that AI will tend to have goals that are bad for humans
There are many forms of power. It is unclear that a competence advantage will ultimately trump all others in time
This argument also appears to apply to human groups such as corporations, so we need an explanation of why those are not an existential risk
People who have favorably discussed[2] this argument (specific quotes here): Paul Christiano (2021), Ajeya Cotra (2023), Eliezer Yudkowsky (2024), Nick Bostrom (2014[3]).
See also: Full wiki page on the competent non-aligned agents argument
Second species argument
Summary:
1. Human dominance over other animal species is primarily due to humans having superior cognitive and coordination abilities
2. Therefore if another 'species' appears with abilities superior to those of humans, that species will become dominant over humans in the same way
3. AI will essentially be a 'species' with superior abilities to humans
4. Therefore AI will dominate humans
Selected counterarguments:
Human dominance over other species is plausibly not due to the cognitive abilities of individual humans, but rather because of human ability to communicate and store information through culture and artifacts
Intelligence in animals doesn't appear to generally relate to dominance. For instance, elephants are much more intelligent than beetles, and it is not clear that elephants have dominated beetles
Differences in capabilities don't necessarily lead to extinction. In the modern world, more powerful countries arguably control less powerful countries, but they do not wipe them out and most colonized countries have eventually gained independence
People who have favorably discussed this argument (specific quotes here): Joe Carlsmith (2024), Richard Ngo (2020), Stuart Russell (2020[4]), Nick Bostrom (2015).
See also: Full wiki page on the second species argument
Loss of control via inferiority
Summary:
1. AI systems will become much more competent than humans at decision-making
2. Thus most decisions will probably be allocated to AI systems
3. If AI systems make most decisions, humans will lose control of the future
4. If humans have no control of the future, the future will probably be bad for humans
Selected counterarguments:
Humans do not generally seem to become disempowered by possession of software that is far superior to them, even if it makes many 'decisions' in the process of carrying out their will
In the same way that humans avoid being overpowered by companies, even though companies are more competent than individual humans, humans can track AI trustworthiness and have AI systems compete for them as users. This might substantially mitigate untrustworthy AI behavior
People who have favorably discussed this argument (specific quotes here): Paul Christiano (2014), Ajeya Cotra (2023), Richard Ngo (2024).
See also: Full wiki page on loss of control via inferiority
Loss of control via speed
Summary:
1. Advances in AI will produce...