Agency Emerges From Optimization

AGI timelines, offense/defense balance, evolution vs engineering, how to lower P(Doom), Eliezer Yudkowksy, and much more!

Timestamps:

00:00 Trailer

03:10 Is My P(Doom) Lowering?

11:29 First Caller: AI Offense vs Defense Balance

16:50 Superintelligence Skepticism

25:05 Agency and AI Goals

29:06 Communicating AI Risk

36:35 Attack vs Defense Equilibrium

38:22 Can We Solve Outer Alignment?

54:47 What is Your P(Pocket Nukes)?

1:00:05 The “Shoggoth” Metaphor Is Outdated

1:06:23 Should I Reframe the P(Doom) Question?

1:12:22 How YOU Can Make a Difference

1:24:43 Can AGI Beat Biology?

1:39:22 Agency and Convergent Goals

1:59:56 Viewer Poll: What Content Should I Make?

2:26:15 AI Warning Shots

2:32:12 More Listener Questions: Debate Tactics, Getting a PhD, Specificity

2:53:53 Closing Thoughts

Links:

Support PauseAI — https://pauseai.info/

Support PauseAI US — https://www.pauseai-us.org/

Support LessWrong / Lightcone Infrastructure — LessWrong is fundraising!

Support MIRI — MIRI’s 2025 Fundraiser

About the show:

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏