The Nonlinear Library EA - Summary of Situational Awareness - The Decade Ahead by OscarD
Jun 8, 2024
30:29
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Summary of Situational Awareness - The Decade Ahead, published by OscarD on June 8, 2024 on The Effective Altruism Forum.
Original by Leopold Aschenbrenner, this summary is not commissioned or endorsed by him.
Short Summary
Extrapolating existing trends in compute, spending, algorithmic progress, and energy needs implies AGI (remote jobs being completely automatable) by ~2027.
AGI will greatly accelerate AI research itself, leading to vastly superhuman intelligences being created ~1 year after AGI.
Superintelligence will confer a decisive strategic advantage militarily by massively accelerating all spheres of science and technology.
Electricity use will be a bigger bottleneck on scaling datacentres than investment, but is still doable domestically in the US by using natural gas.
AI safety efforts in the US will be mostly irrelevant if other actors steal the model weights of an AGI. US AGI research must employ vastly better cybersecurity, to protect both model weights and algorithmic secrets.
Aligning superhuman AI systems is a difficult technical challenge, but probably doable, and we must devote lots of resources towards this.
China is still competitive in the AGI race, and China being first to superintelligence would be very bad because it may enable a stable totalitarian world regime. So the US must win to preserve a liberal world order.
Within a few years both the CCP and USG will likely 'wake up' to the enormous potential and nearness of superintelligence, and devote massive resources to 'winning'.
USG will nationalise AGI R&D to improve security and avoid secrets being stolen, and to prevent unconstrained private actors from becoming the most powerful players in the world.
This means much of existing AI governance work focused on AI company regulations is missing the point, as AGI will soon be nationalised.
This is just one story of how things could play out, but a very plausible and scarily soon and dangerous one.
I. From GPT-4 to AGI: Counting the OOMs
Past AI progress
Increases in 'effective compute' have led to consistent increases in model performance over several years and many orders of magnitude (OOMs)
GPT-2 was akin to roughly a preschooler level of intelligence (able to piece together basic sentences sometimes), GPT-3 at the level of an elementary schooler (able to do some simple tasks with clear instructions), and GPT-4 similar to a smart high-schooler (able to write complicated functional code, long coherent essays, and answer somewhat challenging maths questions).
Superforecasters and experts have consistently underestimated future improvements in model performance, for instance:
The creators of the MATH benchmark expected that "to have more traction on mathematical problem solving we will likely need new algorithmic advancements from the broader research community". But within a year of the benchmark's release, state-of-the-art (SOTA) models went from 5% to 50% accuracy, and are now above 90%.
Professional forecasts made in August 2021 expected the MATH benchmark score of SOTA models to be 12.7% in June 2022, but the actual score was 50%.
Experts like Yann LeCun and Gary Marcus have falsely predicted that deep learning will plateau.
Bryan Caplan is on track to lose a public bet for the first time ever after GPT-4 got an A on his economics exam just two months after he bet no AI could do this by 2029.
We can decompose recent progress into three main categories:
Compute: GPT-2 was trained in 2019 with an estimated 4e21 FLOP, and GPT-4 was trained in 2023 with an estimated 8e24 to 4e25 FLOP.[1] This is both because of hardware improvements (Moore's Law) and increases in compute budgets for training runs. Adding more compute at test-time, e.g. by running many copies of an AI, to allow for debate and delegation between each instance, could further boos...
Original by Leopold Aschenbrenner, this summary is not commissioned or endorsed by him.
Short Summary
Extrapolating existing trends in compute, spending, algorithmic progress, and energy needs implies AGI (remote jobs being completely automatable) by ~2027.
AGI will greatly accelerate AI research itself, leading to vastly superhuman intelligences being created ~1 year after AGI.
Superintelligence will confer a decisive strategic advantage militarily by massively accelerating all spheres of science and technology.
Electricity use will be a bigger bottleneck on scaling datacentres than investment, but is still doable domestically in the US by using natural gas.
AI safety efforts in the US will be mostly irrelevant if other actors steal the model weights of an AGI. US AGI research must employ vastly better cybersecurity, to protect both model weights and algorithmic secrets.
Aligning superhuman AI systems is a difficult technical challenge, but probably doable, and we must devote lots of resources towards this.
China is still competitive in the AGI race, and China being first to superintelligence would be very bad because it may enable a stable totalitarian world regime. So the US must win to preserve a liberal world order.
Within a few years both the CCP and USG will likely 'wake up' to the enormous potential and nearness of superintelligence, and devote massive resources to 'winning'.
USG will nationalise AGI R&D to improve security and avoid secrets being stolen, and to prevent unconstrained private actors from becoming the most powerful players in the world.
This means much of existing AI governance work focused on AI company regulations is missing the point, as AGI will soon be nationalised.
This is just one story of how things could play out, but a very plausible and scarily soon and dangerous one.
I. From GPT-4 to AGI: Counting the OOMs
Past AI progress
Increases in 'effective compute' have led to consistent increases in model performance over several years and many orders of magnitude (OOMs)
GPT-2 was akin to roughly a preschooler level of intelligence (able to piece together basic sentences sometimes), GPT-3 at the level of an elementary schooler (able to do some simple tasks with clear instructions), and GPT-4 similar to a smart high-schooler (able to write complicated functional code, long coherent essays, and answer somewhat challenging maths questions).
Superforecasters and experts have consistently underestimated future improvements in model performance, for instance:
The creators of the MATH benchmark expected that "to have more traction on mathematical problem solving we will likely need new algorithmic advancements from the broader research community". But within a year of the benchmark's release, state-of-the-art (SOTA) models went from 5% to 50% accuracy, and are now above 90%.
Professional forecasts made in August 2021 expected the MATH benchmark score of SOTA models to be 12.7% in June 2022, but the actual score was 50%.
Experts like Yann LeCun and Gary Marcus have falsely predicted that deep learning will plateau.
Bryan Caplan is on track to lose a public bet for the first time ever after GPT-4 got an A on his economics exam just two months after he bet no AI could do this by 2029.
We can decompose recent progress into three main categories:
Compute: GPT-2 was trained in 2019 with an estimated 4e21 FLOP, and GPT-4 was trained in 2023 with an estimated 8e24 to 4e25 FLOP.[1] This is both because of hardware improvements (Moore's Law) and increases in compute budgets for training runs. Adding more compute at test-time, e.g. by running many copies of an AI, to allow for debate and delegation between each instance, could further boos...