The Nonlinear Library EA - A Gentle Introduction to Risk Frameworks Beyond Forecasting by pending survival
Apr 11, 2024
58:31
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Gentle Introduction to Risk Frameworks Beyond Forecasting, published by pending survival on April 11, 2024 on The Effective Altruism Forum.
This was originally posted on Nathaniel's and Nuno's substacks (Pending Survival and Forecasting Newsletter, respectively). Subscribe here and here!
Discussion is also occurring on the LessWrong here (couldn't link the posts properly for technical reasons).
Introduction
When the Effective Altruism, Bay Area rationality, judgemental forecasting, and prediction markets communities think about risk, they typically do so along rather idiosyncratic and limited lines. These overlook relevant insights and practices from related expert communities, including the fields of disaster risk reduction, safety science, risk analysis, science and technology studies - like the sociology of risk - and futures studies.
To remedy this state of affairs, this document - written by Nathaniel Cooke and edited by Nuño Sempere - (1) explains how disaster risks are conceptualised by risk scholars, (2) outlines Normal Accident Theory and introduces the concept of high-reliability organisations, (3) summarises the differences between "sexy" and "unsexy" global catastrophic risk (GCR) scenarios, and (4) provides a quick overview of the methods professionals use to study the future.
This is not a comprehensive overview, but rather a gentle introduction.
Risk has many different definitions, but this document works with the IPCC definition of the "potential for adverse consequences", where risk is a function of the magnitude of the consequences and the uncertainty around those consequences, recognising a diversity of values and objectives.1 Scholars vary on whether it is always possible to measure uncertainty, but there is a general trend to assume that some uncertainty is so extreme as to be practically unquantifiable.2 Uncertainty here can
reflect both objective likelihood and subjective epistemic uncertainty.
1. Disaster Risk Models
A common saying in disaster risk circles is that "there is no such thing as a natural disaster". As they see it, hazards may arise from nature, but an asteroid striking Earth is only able to threaten humanity because our societies currently rely on vulnerable systems that an asteroid could disrupt or destroy.3
This section will focus on how disaster risk reduction scholars break risks down into their components, model the relationship between disasters and their root causes, structure the process of risk reduction, and conceptualise the biggest and most complex of the risks they study.
The field of global catastrophic risk, in which EAs and adjacent communities are particularly interested, has incorporated some insights from disaster risk reduction. For example, in past disasters, states have often justified instituting authoritarian emergency powers in response to the perceived risks of mass panic, civil disorder and violence, or helplessness among the public.
However, disaster risk scholarship has shown these fears to be baseless.4-9 and this has been integrated into GCR research under the concept of the "Stomp Reflex".
However, other insights from disaster risk scholarship remain neglected, so we encourage readers to consider how they might apply within their cause area and interests.
1.1 The Determinants of Risk
Popular media often refers to things like nuclear war, climate change, and lethal autonomous weapons systems as "risks" in themselves. This stands in contrast with how risk scholars typically think about risk. To these scholars, "risk" refers to outcomes - the "potential for adverse consequences" - rather than the causes of those outcomes. So what would a disaster risk expert consider a nuclear explosion to be if not a risk, and why does it matter?
There is no universal standard model of disaster risk. However, the v...
This was originally posted on Nathaniel's and Nuno's substacks (Pending Survival and Forecasting Newsletter, respectively). Subscribe here and here!
Discussion is also occurring on the LessWrong here (couldn't link the posts properly for technical reasons).
Introduction
When the Effective Altruism, Bay Area rationality, judgemental forecasting, and prediction markets communities think about risk, they typically do so along rather idiosyncratic and limited lines. These overlook relevant insights and practices from related expert communities, including the fields of disaster risk reduction, safety science, risk analysis, science and technology studies - like the sociology of risk - and futures studies.
To remedy this state of affairs, this document - written by Nathaniel Cooke and edited by Nuño Sempere - (1) explains how disaster risks are conceptualised by risk scholars, (2) outlines Normal Accident Theory and introduces the concept of high-reliability organisations, (3) summarises the differences between "sexy" and "unsexy" global catastrophic risk (GCR) scenarios, and (4) provides a quick overview of the methods professionals use to study the future.
This is not a comprehensive overview, but rather a gentle introduction.
Risk has many different definitions, but this document works with the IPCC definition of the "potential for adverse consequences", where risk is a function of the magnitude of the consequences and the uncertainty around those consequences, recognising a diversity of values and objectives.1 Scholars vary on whether it is always possible to measure uncertainty, but there is a general trend to assume that some uncertainty is so extreme as to be practically unquantifiable.2 Uncertainty here can
reflect both objective likelihood and subjective epistemic uncertainty.
1. Disaster Risk Models
A common saying in disaster risk circles is that "there is no such thing as a natural disaster". As they see it, hazards may arise from nature, but an asteroid striking Earth is only able to threaten humanity because our societies currently rely on vulnerable systems that an asteroid could disrupt or destroy.3
This section will focus on how disaster risk reduction scholars break risks down into their components, model the relationship between disasters and their root causes, structure the process of risk reduction, and conceptualise the biggest and most complex of the risks they study.
The field of global catastrophic risk, in which EAs and adjacent communities are particularly interested, has incorporated some insights from disaster risk reduction. For example, in past disasters, states have often justified instituting authoritarian emergency powers in response to the perceived risks of mass panic, civil disorder and violence, or helplessness among the public.
However, disaster risk scholarship has shown these fears to be baseless.4-9 and this has been integrated into GCR research under the concept of the "Stomp Reflex".
However, other insights from disaster risk scholarship remain neglected, so we encourage readers to consider how they might apply within their cause area and interests.
1.1 The Determinants of Risk
Popular media often refers to things like nuclear war, climate change, and lethal autonomous weapons systems as "risks" in themselves. This stands in contrast with how risk scholars typically think about risk. To these scholars, "risk" refers to outcomes - the "potential for adverse consequences" - rather than the causes of those outcomes. So what would a disaster risk expert consider a nuclear explosion to be if not a risk, and why does it matter?
There is no universal standard model of disaster risk. However, the v...