Dr. Keith Duggar from Machine Learning Street Talk was the subject of my recent reaction episode about whether GPT o1 can reason. But instead of ignoring or blocking me, Keith was brave enough to come into the lion’s den and debate his points with me… and his P(doom) might shock you!

First we debate whether Keith’s distinction between Turing Machines and Discrete Finite Automata is useful for understanding limitations of current LLMs. Then I take Keith on a tour of alignment, orthogonality, instrumental convergence, and other popular stations on the “doom train”, to compare our views on each.

Keith was a great sport and I think this episode is a classic!

00:00 Introduction

00:46 Keith’s Background

03:02 Keith’s P(doom)

14:09 Are LLMs Turing Machines?

19:09 Liron Concedes on a Point!

21:18 Do We Need >1MB of Context?

27:02 Examples to Illustrate Keith’s Point

33:56 Is Terence Tao a Turing Machine?

38:03 Factoring Numbers: Human vs. LLM

53:24 Training LLMs with Turing-Complete Feedback

1:02:22 What Does the Pillar Problem Illustrate?

01:05:40 Boundary between LLMs and Brains

1:08:52 The 100-Year View

1:18:29 Intelligence vs. Optimization Power

1:23:13 Is Intelligence Sufficient To Take Over?

01:28:56 The Hackable Universe and AI Threats

01:31:07 Nuclear Extinction vs. AI Doom

1:33:16 Can We Just Build Narrow AI?

01:37:43 Orthogonality Thesis and Instrumental Convergence

01:40:14 Debating the Orthogonality Thesis

02:03:49 The Rocket Alignment Problem

02:07:47 Final Thoughts

Show Notes

Keith’s show: https://www.youtube.com/@MachineLearningStreetTalk

Keith’s Twitter: https://x.com/doctorduggar

Keith’s fun brain teaser that LLMs can’t solve yet, about a pillar with four holes: https://youtu.be/nO6sDk6vO0g?si=diGUY7jW4VFsV0TJ&t=3684

Eliezer Yudkowsky’s classic post about the “Rocket Alignment Problem”: https://intelligence.org/2018/10/03/rocket-alignment/

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates.

📣 You can now chat with me and other listeners in the #doom-debates channel of the PauseAI discord: https://discord.gg/2XXWXvErfA

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit lironshapira.substack.com