The Nonlinear Library LW - What's important in "AI for epistemics"? by Lukas Finnveden
Aug 24, 2024
54:05
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What's important in "AI for epistemics"?, published by Lukas Finnveden on August 24, 2024 on LessWrong.
Summary
This post gives my personal take on "AI for epistemics" and how important it might be to work on.
Some background context:
AI capabilities are advancing rapidly and I think it's important to think ahead and prepare for the possible development of AI that could automate almost all economically relevant tasks that humans can do.[1]
That kind of AI would have a huge impact on key epistemic processes in our society. (I.e.: It would have a huge impact on how new facts get found, how new research gets done, how new forecasts get made, and how all kinds of information spread through society.)
I think it's very important for our society to have excellent epistemic processes. (I.e.: For important decisions in our society to be made by people or AI systems who have informed and unbiased beliefs that take into account as much of the available evidence as is practical.)
Accordingly, I'm interested in affecting the development and usage of AI technology in ways that lead towards better epistemic processes.
So: How can we affect AI to contribute to better epistemic processes? When looking at concrete projects, here, I find it helpful to distinguish between two different categories of work:
1.
Working to increase AIs' epistemic capabilities, and in particular, differentially advancing them compared to other AI capabilities. Here, I also include technical work to measure AIs' epistemic capabilities.[2]
2. Efforts to enable the diffusion and appropriate trust of AI-discovered information. This is focused on social dynamics that could cause AI-produced information to be insufficiently or excessively trusted. It's also focused on AIs' role in communicating information (as opposed to just producing it).
Examples of interventions, here, include "create an independent organization that evaluates popular AIs' truthfulness", or "work for countries to adopt good (and avoid bad) legislation of AI communication".
I'd be very excited about thoughtful and competent efforts in this second category. However, I talk significantly more about efforts in the first category, in this post. This is just an artifact of how this post came to be, historically - it's not because I think work on the second category of projects is less important.[3]
For the first category of projects: Technical projects to differentially advance epistemic capabilities seem somewhat more "shovel-ready". Here, I'm especially excited about projects that differentially boost AI epistemic capabilities in a manner that's some combination of durable and/or especially good at demonstrating those capabilities to key actors.
Durable means that projects should (i) take the bitter lesson into account by working on problems that won't be solved-by-default when more compute is available, and (ii) work on problems that industry isn't already incentivized to put huge efforts into (such as "making AIs into generally better agents"). (More on these criteria here.)
Two example projects that I think fulfill these criteria (I discuss a lot more projects here):
Experiments on what sort of arguments and decompositions make it easier for humans to reach the truth in hard-to-verify areas. (Strongly related to scalable oversight.)
Using AI to generate large quantities of forecasting data, such as by automatically generating and resolving questions.
Separately, I think there's value in demonstrating the potential of AI epistemic advice to key actors - especially frontier AI companies and governments. When transformative AI (TAI)[4] is first developed, it seems likely that these actors will (i) have a big advantage in their ability to accelerate AI-for-epistemics via their access to frontier models and algorithms, and (ii) that I especially car...
Summary
This post gives my personal take on "AI for epistemics" and how important it might be to work on.
Some background context:
AI capabilities are advancing rapidly and I think it's important to think ahead and prepare for the possible development of AI that could automate almost all economically relevant tasks that humans can do.[1]
That kind of AI would have a huge impact on key epistemic processes in our society. (I.e.: It would have a huge impact on how new facts get found, how new research gets done, how new forecasts get made, and how all kinds of information spread through society.)
I think it's very important for our society to have excellent epistemic processes. (I.e.: For important decisions in our society to be made by people or AI systems who have informed and unbiased beliefs that take into account as much of the available evidence as is practical.)
Accordingly, I'm interested in affecting the development and usage of AI technology in ways that lead towards better epistemic processes.
So: How can we affect AI to contribute to better epistemic processes? When looking at concrete projects, here, I find it helpful to distinguish between two different categories of work:
1.
Working to increase AIs' epistemic capabilities, and in particular, differentially advancing them compared to other AI capabilities. Here, I also include technical work to measure AIs' epistemic capabilities.[2]
2. Efforts to enable the diffusion and appropriate trust of AI-discovered information. This is focused on social dynamics that could cause AI-produced information to be insufficiently or excessively trusted. It's also focused on AIs' role in communicating information (as opposed to just producing it).
Examples of interventions, here, include "create an independent organization that evaluates popular AIs' truthfulness", or "work for countries to adopt good (and avoid bad) legislation of AI communication".
I'd be very excited about thoughtful and competent efforts in this second category. However, I talk significantly more about efforts in the first category, in this post. This is just an artifact of how this post came to be, historically - it's not because I think work on the second category of projects is less important.[3]
For the first category of projects: Technical projects to differentially advance epistemic capabilities seem somewhat more "shovel-ready". Here, I'm especially excited about projects that differentially boost AI epistemic capabilities in a manner that's some combination of durable and/or especially good at demonstrating those capabilities to key actors.
Durable means that projects should (i) take the bitter lesson into account by working on problems that won't be solved-by-default when more compute is available, and (ii) work on problems that industry isn't already incentivized to put huge efforts into (such as "making AIs into generally better agents"). (More on these criteria here.)
Two example projects that I think fulfill these criteria (I discuss a lot more projects here):
Experiments on what sort of arguments and decompositions make it easier for humans to reach the truth in hard-to-verify areas. (Strongly related to scalable oversight.)
Using AI to generate large quantities of forecasting data, such as by automatically generating and resolving questions.
Separately, I think there's value in demonstrating the potential of AI epistemic advice to key actors - especially frontier AI companies and governments. When transformative AI (TAI)[4] is first developed, it seems likely that these actors will (i) have a big advantage in their ability to accelerate AI-for-epistemics via their access to frontier models and algorithms, and (ii) that I especially car...