AI's Evolutionary Perils

Today I’m sharing my interview on Robert Wright’s Nonzero Podcast from last May. Rob is an especially sharp interviewer who doesn't just nod along, he had great probing questions for me.

This interview happened right after Ilya Sutskever and Jan Leike resigned from OpenAI in May 2024, continuing a pattern that goes back to Dario Amodei leaving to start Anthropic. These aren't fringe doomers; these are the people hired specifically to solve the safety problem, and they keep concluding it's not solvable at the current pace.

00:00:00 - Liron’s preface

00:02:10 - Robert Wright introduces Liron

00:04:02 - PauseAI protests at OpenAI headquarters

00:05:15 - OpenAI resignations (Ilya Sutskever, Jan Leike, Dario Amodei, Paul Christiano, Daniel Kokotajlo)

00:15:30 - P vs NP problem as analogy for AI alignment difficulty

00:22:31 - AI pause movement and protest turnout

00:29:02 - Defining AI doom and sci-fi scenarios

00:32:05 - What’s My P(Doom)™

00:35:18 - Fast vs slow AI takeoff and Sam Altman's position

00:42:33 - Paperclip thought experiment and instrumental convergence explanation

00:54:40 - Concrete examples of AI power-seeking behavior (business assistant scenario)

01:00:58 - GPT-4 TaskRabbit deception example and AI reasoning capabilities

01:09:00 - AI alignment challenges and human values discussion

01:17:33 - Wrap-up and transition to premium subscriber content

Show Notes

This episode on Rob’s Nonzero Newsletter. You can subscribe for premium access to the last 1 hour of our discussion! — https://www.nonzero.org/p/in-defense-of-ai-doomerism-robert

This episode on Rob’s YouTube — https://www.youtube.com/watch?v=VihA_-8kBNg

PauseAI — https://pauseai.info

PauseAI US — http://pauseai-us.org

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates