The Nonlinear Library LW - List your AI X-Risk cruxes! by Aryeh Englander
Apr 29, 2024
02:52
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: List your AI X-Risk cruxes!, published by Aryeh Englander on April 29, 2024 on LessWrong.
[I'm posting this as a very informal community request in lieu of a more detailed writeup, because if I wait to do this in a much more careful fashion then it probably won't happen at all. If someone else wants to do a more careful version that would be great!]
By crux here I mean some uncertainty you have such that your estimate for the likelihood of existential risk from AI - your "p(doom)" if you like that term - might shift significantly if that uncertainty were resolved.
More precisely, let's define a crux as a proposition such that: (a) your estimate for the likelihood of existential catastrophe due to AI would shift a non-trivial amount depending on whether that proposition was true or false; (b) you think there's at least a non-trivial probability that the proposition is true; and (c) you also think there's at least a non-trivial probability that the proposition is false.
Note 1: It could also be a variable rather than a binary proposition, for example "year human-level AGI is achieved". In that case substitute "variable is above some number x" and "variable is below some number y" instead of proposition is true / proposition is false.
Note 2: It doesn't have to be that the proposition / variable on it's own would significantly shift your estimate. If some combination of propositions / variables would shift your estimate, then those propositions / variables are cruxes at least when combined.
For concreteness let's say that "non-trivial" here means at least 5%. So you need to think there's at least a 5% chance the proposition is true, and at least a 5% chance that it's false, and also that your estimate for p(existential catastrophe due to AI) would shift by at least 5% depending on whether the proposition is true or false.
Here are just a few examples of potential cruxes people might have (among many others!):
Year human-level AGI is achieved
How fast the transition will be from much lower-capability AI to roughly human-level AGI, or from roughly human-level AGI to vastly superhuman AI
Whether power seeking will be an instrumentally convergent goal
Whether AI will greatly upset the offense-defense balance for CBRN technologies in a way that favors malicious actors
Whether AGIs could individually or collectively defeat humanity if they wanted to
Whether the world can collectively get their collective act together to pause AGI development given a clear enough signal (in combination with the probability that we'll in fact get a clear enough signal in time
Listing all your cruxes would be the most useful, but if that is too long a list then just list the ones you find most important. Providing additional details (for example, your probability distribution for each crux and/or how exactly it would shift your p(doom) estimates) is recommended if you can but isn't necessary.
Commenting with links to other related posts on LW or elsewhere might be useful as well.
Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
[I'm posting this as a very informal community request in lieu of a more detailed writeup, because if I wait to do this in a much more careful fashion then it probably won't happen at all. If someone else wants to do a more careful version that would be great!]
By crux here I mean some uncertainty you have such that your estimate for the likelihood of existential risk from AI - your "p(doom)" if you like that term - might shift significantly if that uncertainty were resolved.
More precisely, let's define a crux as a proposition such that: (a) your estimate for the likelihood of existential catastrophe due to AI would shift a non-trivial amount depending on whether that proposition was true or false; (b) you think there's at least a non-trivial probability that the proposition is true; and (c) you also think there's at least a non-trivial probability that the proposition is false.
Note 1: It could also be a variable rather than a binary proposition, for example "year human-level AGI is achieved". In that case substitute "variable is above some number x" and "variable is below some number y" instead of proposition is true / proposition is false.
Note 2: It doesn't have to be that the proposition / variable on it's own would significantly shift your estimate. If some combination of propositions / variables would shift your estimate, then those propositions / variables are cruxes at least when combined.
For concreteness let's say that "non-trivial" here means at least 5%. So you need to think there's at least a 5% chance the proposition is true, and at least a 5% chance that it's false, and also that your estimate for p(existential catastrophe due to AI) would shift by at least 5% depending on whether the proposition is true or false.
Here are just a few examples of potential cruxes people might have (among many others!):
Year human-level AGI is achieved
How fast the transition will be from much lower-capability AI to roughly human-level AGI, or from roughly human-level AGI to vastly superhuman AI
Whether power seeking will be an instrumentally convergent goal
Whether AI will greatly upset the offense-defense balance for CBRN technologies in a way that favors malicious actors
Whether AGIs could individually or collectively defeat humanity if they wanted to
Whether the world can collectively get their collective act together to pause AGI development given a clear enough signal (in combination with the probability that we'll in fact get a clear enough signal in time
Listing all your cruxes would be the most useful, but if that is too long a list then just list the ones you find most important. Providing additional details (for example, your probability distribution for each crux and/or how exactly it would shift your p(doom) estimates) is recommended if you can but isn't necessary.
Commenting with links to other related posts on LW or elsewhere might be useful as well.
Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org