Future of Life Institute Podcast Why Building Superintelligence Means Human Extinction (with Nate Soares)
15 snips
Sep 18, 2025 
Nate Soares, President of the Machine Intelligence Research Institute and co-author of "If Anyone Builds It, Everyone Dies," dives into the urgent risks posed by advanced AI systems. He explains how current AI is 'grown, not crafted,' leading to unpredictable behavior. Soares highlights the peril of intelligence threshold effects and the dangers of a failed superintelligence deployment, which lacks second chances. He advocates for an international ban on superintelligence research to mitigate existential risks, stressing that humanity's current responses are insufficient.
AI Snips
Chapters
Books
Transcript
Episode notes
Hidden Alarmism Blocks Action
- Nate Soares argues society is alarmed about AI but reluctant to admit it publicly, creating paralysis.
- He wrote the book to break that logjam and spur serious mainstream action.
Transformer Shift Happened Fast
- Soares compares the pace of ML progress to sudden paradigm shifts like transformers enabling ChatGPT in years.
- He warns forecasting decades using past architectures misses imminent breakthroughs.
Thresholds Make Small Gains Dangerous
- Intelligence often shows threshold effects where small changes produce huge real-world impact.
- A modest scale-up could push models past a critical point and change the game rapidly.
