The Nonlinear Library EA - My Current Claims and Cruxes on LLM Forecasting & Epistemics by Ozzie Gooen
Jun 27, 2024
39:34
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My Current Claims and Cruxes on LLM Forecasting & Epistemics, published by Ozzie Gooen on June 27, 2024 on The Effective Altruism Forum.
I think that recent improvements in LLMs have brought us to the point where LLM epistemic systems are starting to be useful. After spending some time thinking about it, I've realized that such systems, broadly, seem very promising to me as an effective altruist intervention area. However, I think that our community has yet to do a solid job outlining what this area could look like or figuring out key uncertainties.
This document presents a rough brainstorm on these topics. While I could dedicate months to refining these ideas, I've chosen to share these preliminary notes now to spark discussion. If you find the style too terse, feel free to use an LLM to rewrite it in a format you prefer.
I believe my vision for this area is more ambitious and far-reaching (i.e. not narrow to a certain kind of forecasting) than what I've observed in other discussions. I'm particularly excited about AI-heavy epistemic improvements, which I believe have greater potential than traditional forecasting innovations.
I'm trying to figure out what to make of this regarding our future plans at QURI, and I recommend that other organizations in the space consider similar updates.
Key Definitions:
Epistemic process: A set of procedures to do analysis work, often about topics with a lot of uncertainty. This could be "have one journalist do everything themselves", to a complex (but repeatable) ecosystem of multiple humans and software systems.
LLM-based Epistemic Process (LEP): A system that relies on LLMs to carry out most or all of an epistemic process. This might start at ~10% LLM-labor, but can gradually ramp up. I imagine that such a process is likely to feature some kinds of estimates or forecasts.
Scaffolding: Software used around an LLM system, often to make it valuable for specific use-cases. In the case of an LEP, a lot of scaffolding might be needed.
1. High-Level Benefits & Uses
Claim 1: If humans could forecast much better, these humans should make few foreseeable mistakes. This covers many mistakes, particularly ones we might be worried about now.
Someone deciding about talking to a chatbot that can be predicted to be net-negative (perhaps it would create an unhealthy relationship) could see this forecast and simply decide not to start the chat.
Say that a person's epistemic state could follow one of four trajectories, depending on some set of reading materials. For example, one set is conspiratorial, one is religious, etc. Good forecasting could help forecast this and inform the person ahead of time.
Note that this can be radical and perhaps dangerous. For example, a religious family knowing how to keep their children religious with a great deal of certainty.
Say that one of two political candidates is predictably terrible. This could be made clear to voters who trust said prediction.
If an AI actor is doing something likely to be monopolistic or dangerous, this would be made more obvious to itself and those around it.
Note: There will also be unforeseeable mistakes, but any actions that we could do that are foreseeably-high-value for them, could be predicted. For example, general-purpose risk mitigation measures.
Claim 2: Highly intelligent / epistemically capable organizations are likely to be better at coordination.
This might well mean fewer wars and conflict, along with corresponding military spending.
If highly capable actors were in a prisoner's dilemma, the results could be ugly. But very often, there's a lot of potential and value in not getting into one in the first place.
Evidence: From The Better Angels of Our Nature, there's significant evidence that humanity has become significantly less violent over time. One potential exception is t...
I think that recent improvements in LLMs have brought us to the point where LLM epistemic systems are starting to be useful. After spending some time thinking about it, I've realized that such systems, broadly, seem very promising to me as an effective altruist intervention area. However, I think that our community has yet to do a solid job outlining what this area could look like or figuring out key uncertainties.
This document presents a rough brainstorm on these topics. While I could dedicate months to refining these ideas, I've chosen to share these preliminary notes now to spark discussion. If you find the style too terse, feel free to use an LLM to rewrite it in a format you prefer.
I believe my vision for this area is more ambitious and far-reaching (i.e. not narrow to a certain kind of forecasting) than what I've observed in other discussions. I'm particularly excited about AI-heavy epistemic improvements, which I believe have greater potential than traditional forecasting innovations.
I'm trying to figure out what to make of this regarding our future plans at QURI, and I recommend that other organizations in the space consider similar updates.
Key Definitions:
Epistemic process: A set of procedures to do analysis work, often about topics with a lot of uncertainty. This could be "have one journalist do everything themselves", to a complex (but repeatable) ecosystem of multiple humans and software systems.
LLM-based Epistemic Process (LEP): A system that relies on LLMs to carry out most or all of an epistemic process. This might start at ~10% LLM-labor, but can gradually ramp up. I imagine that such a process is likely to feature some kinds of estimates or forecasts.
Scaffolding: Software used around an LLM system, often to make it valuable for specific use-cases. In the case of an LEP, a lot of scaffolding might be needed.
1. High-Level Benefits & Uses
Claim 1: If humans could forecast much better, these humans should make few foreseeable mistakes. This covers many mistakes, particularly ones we might be worried about now.
Someone deciding about talking to a chatbot that can be predicted to be net-negative (perhaps it would create an unhealthy relationship) could see this forecast and simply decide not to start the chat.
Say that a person's epistemic state could follow one of four trajectories, depending on some set of reading materials. For example, one set is conspiratorial, one is religious, etc. Good forecasting could help forecast this and inform the person ahead of time.
Note that this can be radical and perhaps dangerous. For example, a religious family knowing how to keep their children religious with a great deal of certainty.
Say that one of two political candidates is predictably terrible. This could be made clear to voters who trust said prediction.
If an AI actor is doing something likely to be monopolistic or dangerous, this would be made more obvious to itself and those around it.
Note: There will also be unforeseeable mistakes, but any actions that we could do that are foreseeably-high-value for them, could be predicted. For example, general-purpose risk mitigation measures.
Claim 2: Highly intelligent / epistemically capable organizations are likely to be better at coordination.
This might well mean fewer wars and conflict, along with corresponding military spending.
If highly capable actors were in a prisoner's dilemma, the results could be ugly. But very often, there's a lot of potential and value in not getting into one in the first place.
Evidence: From The Better Angels of Our Nature, there's significant evidence that humanity has become significantly less violent over time. One potential exception is t...