How close are we to the end of humanity? Toby Ord, Senior Researcher at Oxford University’s AI Governance Initiative and author of The Precipice, argues that the odds of a civilization-ending catastrophe this century are roughly one in six. In this wide-ranging conversation, we unpack the risks that could end humanity’s story and explore why protecting future generations may be our greatest moral duty.
We explore:
• Why existential risk matters and what we owe the 10,000-plus generations who came before us
• Why Toby believes we face a one-in-six chance of civilizational collapse this century
• The four key types of AI risk: alignment failures, gradual disempowerment, AI-fueled coups, and AI-enabled weapons of mass destruction
• Why racing dynamics between companies and nations amplify those risks, and how an AI treaty might help
• How short-term incentives in democracies blind us to century-scale dangers, along with policy ideas to fix it
• The lessons COVID should have taught us (but didn’t)
• The hidden ways the nuclear threat has intensified as treaties lapse and geopolitical tensions rise
• Concrete steps each of us can take today to steer humanity away from the brink
—
Transcript: https://www.generalist.com/p/existential-risk-and-the-future-of-humanity-toby-ord
—
This episode is brought to you by Brex: The banking solution for startups.
—
Timestamps
(00:00) Intro
(02:20) An explanation of existential risk, and the study of it
(06:20) How Toby’s interest in global poverty sparked his founding of Giving What We Can
(11:18) Why Toby chose to study under Derek Parfit at Oxford
(14:40) Population ethics, and how Parfit’s philosophy looked ahead to future generations
(19:05) An introduction to existential risk
(22:40) Why we should care about the continued existence of humans
(28:53) How fatherhood sparked Toby’s gratitude to his parents and previous generations
(31:57) An explanation of how LLMs and agents work
(40:10) The four types of AI risks
(46:58) How humans justify bad choices: lessons from the Manhattan Project
(51:29) A breakdown of the “unilateralist’s curse” and a case for an AI treaty
(1:02:15) Covid’s impact on our understanding of pandemic risk
(1:08:51) The shortcomings of our democracies and ways to combat our short-term focus
(1:14:50) Final meditations
—
Follow Toby Ord
Website: https://www.tobyord.com/
LinkedIn: https://www.linkedin.com/in/tobyord
X: https://x.com/tobyordoxford?lang=en
Giving What We Can: https://www.givingwhatwecan.org/
—
Resources and episode mentions
—Books—
• The Precipice: Existential Risk and the Future of Humanity: https://www.amazon.com/dp/0316484911
• Reasons and Persons: https://www.amazon.com/Reasons-Persons-Derek-Parfit/dp/019824908X
• Practical Ethics: https://www.amazon.com/Practical-Ethics-Peter-Singer/dp/052143971X
—People—
• Derek Parfit: https://en.wikipedia.org/wiki/Derek_Parfit
• Carl Sagan: https://en.wikipedia.org/wiki/Carl_Sagan
• Stuart Russell: https://en.wikipedia.org/wiki/Stuart_J._Russell
—Other resources—
• DeepMind: https://deepmind.google/
• OpenAI: https://openai.com/
• Manhattan Project: https://en.wikipedia.org/wiki/Manhattan_Project
• The Unilateralist’s Curse and the Case for a Principle of Conformity: https://nickbostrom.com/papers/unilateralist.pdf
• The Nuclear Non-Proliferation Treaty (NPT), 1968: https://history.state.gov/milestones/1961-1968/npt
• The Blitz: https://en.wikipedia.org/wiki/The_Blitz
• Operation Warp Speed: https://en.wikipedia.org/wiki/Operation_Warp_Speed
—
Production and marketing by penname.co. For inquiries about sponsoring the podcast, email jordan@penname.co.
