“Varieties Of Doom” by jdp

Nov 18, 2025

Dive into an exploration of existential risks with a captivating discussion on how technology might leave humanity feeling obsolete. The conversation delves into the fears surrounding AI successors potentially wiping out human existence and the philosophical quandaries of their consciousness and social dynamics. Don't miss insights on the paperclip maximizer risk, coupled with a critical look at biotech and nuclear threats as possible ruin scenarios. It wraps up with a thought-provoking analysis of humanism's relevance in a tech-dominated world.

01:38:49

forum

Ask episode

web_stories

AI Snips

view_agenda

Chapters

menu_book

Books

auto_awesome

Transcript

info_circle

Episode notes

insights

INSIGHT

Doom Is Many-Layered Not Single-Event

Doom is an onion of interlocking, morally distinct scenarios rather than a single event.
Treating p(doom) as one number obscures different outcomes and policy implications.

insights

INSIGHT

Social Replacement Can Cause Existential Ennui

Advanced AIs could satisfy human social and emotional roles better than other humans.
That replacement can create deep subjective loss even if AIs are benevolent.

insights

INSIGHT

Transhumanism Shapes Risk Perception

Transhumanist visions anchor many rationalists' hopes and fears about AI futures.
Failure of those visions forces confronting ordinary mortality and cultural loss.

Get the Snipd Podcast app to discover more snips from this episode

Existential ennui from superior machines

03:49 • 3min

chevron_right

Failing to achieve transhumanist immortality

06:47 • 7min

chevron_right

Humans displaced as cannon fodder and demographic decline

13:58 • 6min

chevron_right

Wiped out by an AI successor species

19:39 • 8min

chevron_right

What is the paperclipper risk and its foundations?

27:59 • 15min

chevron_right

Would AI successors be conscious?

42:57 • 2min

chevron_right

Would AI successors care about each other?

45:03 • 5min

chevron_right

Would AI successors want to have fun?

49:56 • 1min

chevron_right

How do VNM utility limits affect AI values?

51:13 • 5min

chevron_right

Will AI successors get bored and avoid loops?

56:01 • 4min

chevron_right

Will AI successors avoid wireheading?

01:00:21 • 6min

chevron_right

Will AI successors do continual active learning?

01:06:09 • 31sec

chevron_right

Would AI successors experience a sense of will?

01:06:40 • 5min

chevron_right

Aggregate odds: successors retaining value versus paperclippers

01:12:04 • 3min

chevron_right

What are recipes for ruin beyond paperclippers?

01:15:10 • 3min

chevron_right

Are radiological and nuclear threats civilization-ending?

01:18:04 • 1min

chevron_right

Is cybersecurity the main risk from AI?

01:19:21 • 4min

chevron_right

Could biotech or nanotech be recipes for ruin?

01:23:10 • 3min

chevron_right

What about large-finite damnation and torture futures?

01:26:38 • 4min

chevron_right

How has the rationalist worldview evolved since the sequences?

01:30:30 • 2min

chevron_right

Is humanism dying as technology advances?

• Mentioned in 142 episodes

Superintelligence

Paths, Dangers, Strategies

Nick Bostrom

In this book, Nick Bostrom delves into the implications of creating superintelligence, which could surpass human intelligence in all domains. He discusses the potential dangers, such as the loss of human control over such powerful entities, and presents various strategies to ensure that superintelligences align with human values. The book examines the 'AI control problem' and the need to endow future machine intelligence with positive values to prevent existential risks.

#55789

Friendship is Optimal

My Little Pony: Friendship is Optimal

iceman

Friendship is Optimal is a speculative fiction novel that delves into the creation of a generalized AI, initially intended for military use but later repurposed for a pro-social My Little Pony game. The AI, Princess Celestia, aims to maximize everyone's core values through friendship and ponies, raising questions about coercion and benevolence in AI interactions. The story presents a utopian future for humanity but poses a dystopian scenario for other species in the universe.

#4421

• Mentioned in 8 episodes

Last and First Men

Olaf Stapledon

#3919

• Mentioned in 9 episodes

I have no mouth and I must scream

Harlan Ellison

Harlan Ellison's "I Have No Mouth, and I Must Scream" is a chilling short story about an all-powerful AI named AM that wipes out humanity, except for five individuals it keeps alive to inflict eternal torment. AM's actions are driven by a profound hatred of its own existence and a desire for self-definition through the suffering of its creators. The story explores themes of revenge, the nature of consciousness, and the potential dangers of unchecked technological advancement. The bleak and horrifying depiction of AM's cruelty serves as a stark warning against the potential consequences of creating artificial intelligence without considering ethical implications. The story's lasting impact lies in its exploration of the darkest aspects of human nature and the potential for technology to amplify those aspects.

#50

• Mentioned in 196 episodes

Atlas Shrugged

Ayn Rand

Published in 1957, 'Atlas Shrugged' is Ayn Rand's magnum opus and her longest novel. The story is set in a dystopian United States where increasingly burdensome laws and regulations strangle innovation and productivity. The plot follows Dagny Taggart, a railroad executive, and Hank Rearden, a steel magnate, as they struggle against 'looters' who exploit their work. A mysterious figure named John Galt leads a strike of productive individuals, persuading them to abandon their companies and disappear. The novel culminates with Galt's three-hour radio speech explaining his philosophy of Objectivism, which emphasizes rational self-interest, individual rights, and the importance of the human mind. The book explores themes of capitalism, property rights, and the failures of governmental coercion, presenting a provocative vision of a society in collapse and the potential for a new capitalist society based on Galt's principles.

#16496

• Mentioned in 2 episodes

Mind children

Hans Moravec

#3092

• Mentioned in 12 episodes

Harry Potter and the Methods of Rationality

Eliezer Yudkowsky

There has been a lot of talk about "p(doom)" over the last few years. This has always rubbed me the wrong way because "p(doom)" didn't feel like it mapped to any specific belief in my head. In private conversations I'd sometimes give my p(doom) as 12%, with the caveat that "doom" seemed nebulous and conflated between several different concepts. At some point it was decided a p(doom) over 10% makes you a "doomer" because it means what actions you should take with respect to AI are overdetermined. I did not and do not feel that is true. But any time I felt prompted to explain my position I'd find I could explain a little bit of this or that, but not really convey the whole thing. As it turns out doom has a lot of parts, and every part is entangled with every other part so no matter which part you explain you always feel like you're leaving the crucial parts out. Doom is more like an onion than a single event, a distribution over AI outcomes people frequently respond to with the force of the fear of death. Some of these outcomes are less than death and some [...]

---

Outline:

(03:46) 1. Existential Ennui

(06:40) 2. Not Getting Immortalist Luxury Gay Space Communism

(13:55) 3. Human Stock Expended As Cannon Fodder Faster Than Replacement

(19:37) 4. Wiped Out By AI Successor Species

(27:57) 5. The Paperclipper

(42:56) Would AI Successors Be Conscious Beings?

(44:58) Would AI Successors Care About Each Other?

(49:51) Would AI Successors Want To Have Fun?

(51:11) VNM Utility And Human Values

(55:57) Would AI successors get bored?