The Nonlinear Library LW - OMMC Announces RIP by Adam Scholl
Apr 1, 2024
04:33
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: OMMC Announces RIP, published by Adam Scholl on April 1, 2024 on LessWrong.
At the Omnicide Machine Manufacturing Corporation, we work tirelessly to ensure an omnicide-free future. That's why we're excited to announce our Responsible Increase Policy (RIP) - our internal protocol for managing any risks that arise as we create increasingly omnicidal machines.
Inspired by the risk-management framework used in gain-of-function virology research, our RIP defines a framework of Omnicidal Ability Levels (OAL), reflecting the precautions we plan to take as we release increasingly dangerous features over time:
The basic idea of the RIP is simple: each time we ship an update which makes our product more lethal, we will pause our efforts for some amount of time, and then revise our policies to be in some sense more "cautious." For example, our RIP contains the following firm commitments:
We aspire to take actions which are broadly good, rather than broadly bad;
We hope to refrain from releasing any fairly omnicidal systems, until first implementing "certain safeguards";
And we intend to refrain from creating any systems which we're quite sure would kill everyone.
That said, we want to acknowledge that even this cautious approach has drawbacks. For example, if our prevention measures are too weak, we risk catastrophe - potentially leading to extreme, knee-jerk regulatory responses, like banning omnicide machines altogether. On the other hand, if our precautions are too conservative, we risk ending up in a situation where someone who isn't us builds one first.
This is a tricky needle to thread. History is rife with examples of countries deciding to heavily restrict, or even outright ban, technologies which they perceive as incredibly dangerous. So we have designed our RIP to tread lightly, and to exemplify a "minimum viable" safety policy - a well-scoped, small set of tests, that labs can feasibly comply with, and that places the least possible restrictions on frontier existential risks.
The Sweet Lesson: Reasoning is Futile
As an omnicide creation and prevention research company, we think it's important to seriously prepare for worlds in which our product ends up getting built. But the central insight of the modern era of gigantic machines - the so-called "Sweet Lesson" - is that it's possible to build incredibly powerful machines without first developing a deep theoretical understanding of how they work.
Indeed, we currently see ourselves as operating under conditions of near-maximal uncertainty. Time and time again, it has proven futile to try to predict the effects of our actions in advance - new capabilities and failure modes often emerge suddenly and unexpectedly, and we understand little about why.
As such, we endeavor to maintain an attitude of radical epistemic humility. In particular, we assume a uniform prior over the difficulty of survival:
For now, this degree of wholesale, fundamental uncertainty seems inescapable. But in the long-run, we do hope to add information to our world-model - and thanks to our Gain of Omnicide research team, we may soon have it.
Gain of Omnicide
Our Gain of Omnicide research effort aims to generate this information by directly developing omnicidal capacity, in order to then learn how we could have done that safely. Moreover, our core research bet at OMMC is that doing this sort of empirical safety research effectively requires access to frontier omnicide machines.
In our view, the space of possible threats from gigantic omnicide machines is simply too vast to be traversed from the armchair alone. That's why our motto is "Show Don't Tell" - we believe that to prevent the danger associated with these machines, we must first create that danger, since only then can we develop techniques to mitigate it.
But this plan only works if our prototype...
At the Omnicide Machine Manufacturing Corporation, we work tirelessly to ensure an omnicide-free future. That's why we're excited to announce our Responsible Increase Policy (RIP) - our internal protocol for managing any risks that arise as we create increasingly omnicidal machines.
Inspired by the risk-management framework used in gain-of-function virology research, our RIP defines a framework of Omnicidal Ability Levels (OAL), reflecting the precautions we plan to take as we release increasingly dangerous features over time:
The basic idea of the RIP is simple: each time we ship an update which makes our product more lethal, we will pause our efforts for some amount of time, and then revise our policies to be in some sense more "cautious." For example, our RIP contains the following firm commitments:
We aspire to take actions which are broadly good, rather than broadly bad;
We hope to refrain from releasing any fairly omnicidal systems, until first implementing "certain safeguards";
And we intend to refrain from creating any systems which we're quite sure would kill everyone.
That said, we want to acknowledge that even this cautious approach has drawbacks. For example, if our prevention measures are too weak, we risk catastrophe - potentially leading to extreme, knee-jerk regulatory responses, like banning omnicide machines altogether. On the other hand, if our precautions are too conservative, we risk ending up in a situation where someone who isn't us builds one first.
This is a tricky needle to thread. History is rife with examples of countries deciding to heavily restrict, or even outright ban, technologies which they perceive as incredibly dangerous. So we have designed our RIP to tread lightly, and to exemplify a "minimum viable" safety policy - a well-scoped, small set of tests, that labs can feasibly comply with, and that places the least possible restrictions on frontier existential risks.
The Sweet Lesson: Reasoning is Futile
As an omnicide creation and prevention research company, we think it's important to seriously prepare for worlds in which our product ends up getting built. But the central insight of the modern era of gigantic machines - the so-called "Sweet Lesson" - is that it's possible to build incredibly powerful machines without first developing a deep theoretical understanding of how they work.
Indeed, we currently see ourselves as operating under conditions of near-maximal uncertainty. Time and time again, it has proven futile to try to predict the effects of our actions in advance - new capabilities and failure modes often emerge suddenly and unexpectedly, and we understand little about why.
As such, we endeavor to maintain an attitude of radical epistemic humility. In particular, we assume a uniform prior over the difficulty of survival:
For now, this degree of wholesale, fundamental uncertainty seems inescapable. But in the long-run, we do hope to add information to our world-model - and thanks to our Gain of Omnicide research team, we may soon have it.
Gain of Omnicide
Our Gain of Omnicide research effort aims to generate this information by directly developing omnicidal capacity, in order to then learn how we could have done that safely. Moreover, our core research bet at OMMC is that doing this sort of empirical safety research effectively requires access to frontier omnicide machines.
In our view, the space of possible threats from gigantic omnicide machines is simply too vast to be traversed from the armchair alone. That's why our motto is "Show Don't Tell" - we believe that to prevent the danger associated with these machines, we must first create that danger, since only then can we develop techniques to mitigate it.
But this plan only works if our prototype...