The Nonlinear Library EA - Updates on the EA catastrophic risk landscape by Benjamin Todd
May 6, 2024
03:43
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Updates on the EA catastrophic risk landscape, published by Benjamin Todd on May 6, 2024 on The Effective Altruism Forum.
Around the end of Feb 2024 I attended the Summit on Existential Risk and EAG: Bay Area (GCRs), during which I did 25+ one-on-ones about the needs and gaps in the EA-adjacent catastrophic risk landscape, and how they've changed.
The meetings were mostly with senior managers or researchers in the field who I think are worth listening to (unfortunately I can't share names). Below is how I'd summarise the main themes in what was said.
If you have different impressions of the landscape, I'd be keen to hear them.
There's been a big increase in the number of people working on AI safety, partly driven by a reallocation of effort (e.g. Rethink Priorities starting an AI policy think tank); and partly driven by new people entering the field after its newfound prominence.
Allocation in the landscape seems more efficient than in the past - it's harder to identify especially neglected interventions, causes, money, or skill-sets. That means it's become more important to choose based on your motivations. That said, here's a few ideas for neglected gaps:
Within AI risk, it seems plausible the community is somewhat too focused on risks from misalignment rather than mis-use or concentration of power.
There's currently very little work going into issues that arise even if AI is aligned, including the
deployment problem, Will MacAskill's "
grand challenges" and Lukas Finnveden's
list of project ideas. If you put significant probability on alignment being solved, some of these could have high importance too; though most are at the stage where they can't absorb a large number of people.
Within these, digital sentience was the hottest topic, but to me it doesn't obviously seem like the most pressing of these other issues. (Though doing field building for digital sentience is among the more shovel ready of these ideas.)
The concrete entrepreneurial idea that came up the most, and seemed most interesting to me, was founding orgs that use AI to improve epistemics / forecasting / decision-making (I have a
draft post on this - comments welcome).
Post-FTX, funding has become even more dramatically concentrated under Open Philanthropy, so finding new donors seems like a much bigger priority than in the past. (It seems plausible to me that $1bn in a foundation independent from OP could be worth several times that amount added to OP.)
In addition, donors have less money than in the past, while the number of opportunities to fund things in AI safety has increased dramatically, which means marginal funding opportunities seem higher value than in the past (as a concrete example,
nuclear security is getting almost no funding).
Both points mean efforts to start new foundations, fundraise and earn to give all seem more valuable compared to a couple of years ago.
Many people mentioned comms as the biggest issue facing both AI safety and EA. EA has been losing its battle for messaging, and AI safety is in danger of losing its too (with both a new powerful anti-regulation tech lobby and the more left-wing AI ethics scene branding it as sci-fi, doomer, cultish and in bed with labs).
People might be neglecting measures that would help in very short timelines (e.g. transformative AI in under 3 years), though that might be because most people are unable to do much in these scenarios.
Right now, directly talking about AI safety seems to get more people in the door than talking about EA, so some community building efforts have switched to that.
There's been a recent influx in junior people interested in AI safety, so it seems plausible the biggest bottleneck again lies with mentoring & management, rather than recruiting more junior people.
Randomly: there seems to have been a trend of former le...
Around the end of Feb 2024 I attended the Summit on Existential Risk and EAG: Bay Area (GCRs), during which I did 25+ one-on-ones about the needs and gaps in the EA-adjacent catastrophic risk landscape, and how they've changed.
The meetings were mostly with senior managers or researchers in the field who I think are worth listening to (unfortunately I can't share names). Below is how I'd summarise the main themes in what was said.
If you have different impressions of the landscape, I'd be keen to hear them.
There's been a big increase in the number of people working on AI safety, partly driven by a reallocation of effort (e.g. Rethink Priorities starting an AI policy think tank); and partly driven by new people entering the field after its newfound prominence.
Allocation in the landscape seems more efficient than in the past - it's harder to identify especially neglected interventions, causes, money, or skill-sets. That means it's become more important to choose based on your motivations. That said, here's a few ideas for neglected gaps:
Within AI risk, it seems plausible the community is somewhat too focused on risks from misalignment rather than mis-use or concentration of power.
There's currently very little work going into issues that arise even if AI is aligned, including the
deployment problem, Will MacAskill's "
grand challenges" and Lukas Finnveden's
list of project ideas. If you put significant probability on alignment being solved, some of these could have high importance too; though most are at the stage where they can't absorb a large number of people.
Within these, digital sentience was the hottest topic, but to me it doesn't obviously seem like the most pressing of these other issues. (Though doing field building for digital sentience is among the more shovel ready of these ideas.)
The concrete entrepreneurial idea that came up the most, and seemed most interesting to me, was founding orgs that use AI to improve epistemics / forecasting / decision-making (I have a
draft post on this - comments welcome).
Post-FTX, funding has become even more dramatically concentrated under Open Philanthropy, so finding new donors seems like a much bigger priority than in the past. (It seems plausible to me that $1bn in a foundation independent from OP could be worth several times that amount added to OP.)
In addition, donors have less money than in the past, while the number of opportunities to fund things in AI safety has increased dramatically, which means marginal funding opportunities seem higher value than in the past (as a concrete example,
nuclear security is getting almost no funding).
Both points mean efforts to start new foundations, fundraise and earn to give all seem more valuable compared to a couple of years ago.
Many people mentioned comms as the biggest issue facing both AI safety and EA. EA has been losing its battle for messaging, and AI safety is in danger of losing its too (with both a new powerful anti-regulation tech lobby and the more left-wing AI ethics scene branding it as sci-fi, doomer, cultish and in bed with labs).
People might be neglecting measures that would help in very short timelines (e.g. transformative AI in under 3 years), though that might be because most people are unable to do much in these scenarios.
Right now, directly talking about AI safety seems to get more people in the door than talking about EA, so some community building efforts have switched to that.
There's been a recent influx in junior people interested in AI safety, so it seems plausible the biggest bottleneck again lies with mentoring & management, rather than recruiting more junior people.
Randomly: there seems to have been a trend of former le...