1.1 Series summary and Table of Contents
This is a two-post series on AI “foom” (this post) and “doom” (next post).
A decade or two ago, it was pretty common to discuss “foom & doom” scenarios, as advocated especially by Eliezer Yudkowsky. In a typical such scenario, a small team would build a system that would rocket (“foom”) from “unimpressive” to “Artificial Superintelligence” (ASI) within a very short time window (days, weeks, maybe months), involving very little compute (e.g. “brain in a box in a basement”), via recursive self-improvement. Absent some future technical breakthrough, the ASI would definitely be egregiously misaligned, without the slightest intrinsic interest in whether humans live or die. The ASI would be born into a world generally much like today's, a world utterly unprepared for this new mega-mind. The extinction of humans (and every other species) would rapidly follow (“doom”). The ASI would then spend [...]
---
Outline:
(00:11) 1.1 Series summary and Table of Contents
(02:35) 1.1.2 Should I stop reading if I expect LLMs to scale to ASI?
(04:50) 1.2 Post summary and Table of Contents
(07:40) 1.3 A far-more-powerful, yet-to-be-discovered, simple(ish) core of intelligence
(10:08) 1.3.1 Existence proof: the human cortex
(12:13) 1.3.2 Three increasingly-radical perspectives on what AI capability acquisition will look like
(14:18) 1.4 Counter-arguments to there being a far-more-powerful future AI paradigm, and my responses
(14:26) 1.4.1 Possible counter: If a different, much more powerful, AI paradigm existed, then someone would have already found it.
(16:33) 1.4.2 Possible counter: But LLMs will have already reached ASI before any other paradigm can even put its shoes on
(17:14) 1.4.3 Possible counter: If ASI will be part of a different paradigm, who cares? It's just gonna be a different flavor of ML.
(17:49) 1.4.4 Possible counter: If ASI will be part of a different paradigm, the new paradigm will be discovered by LLM agents, not humans, so this is just part of the continuous 'AIs-doing-AI-R&D' story like I've been saying
(18:54) 1.5 Training compute requirements: Frighteningly little
(20:34) 1.6 Downstream consequences of new paradigm with frighteningly little training compute
(20:42) 1.6.1 I'm broadly pessimistic about existing efforts to delay AGI
(23:18) 1.6.2 I'm broadly pessimistic about existing efforts towards regulating AGI
(24:09) 1.6.3 I expect that, almost as soon as we have AGI at all, we will have AGI that could survive indefinitely without humans
(25:46) 1.7 Very little R&D separating seemingly irrelevant from ASI
(26:34) 1.7.1 For a non-imitation-learning paradigm, getting to relevant at all is only slightly easier than getting to superintelligence
(31:05) 1.7.2 Plenty of room at the top
(31:47) 1.7.3 What's the rate-limiter?
(33:22) 1.8 Downstream consequences of very little R&D separating 'seemingly irrelevant' from 'ASI'
(33:30) 1.8.1 Very sharp takeoff in wall-clock time
(35:34) 1.8.1.1 But what about training time?
(36:26) 1.8.1.2 But what if we try to make takeoff smoother?
(37:18) 1.8.2 Sharp takeoff even without recursive self-improvement
(38:22) 1.8.2.1 ...But recursive self-improvement could also happen
(40:12) 1.8.3 Next-paradigm AI probably won't be deployed at all, and ASI will probably show up in a world not wildly different from today's
(42:55) 1.8.4 We better sort out technical alignment, sandbox test protocols, etc., before the new paradigm seems even relevant at all, let alone scary
(43:40) 1.8.5 AI-assisted alignment research seems pretty doomed
(45:22) 1.8.6 The rest of AI for AI safety seems
