Razib Khan's Unsupervised Learning Nate Soares: we are doomed (probably)
Nov 4, 2025
Nate Soares, President of the Machine Intelligence Research Institute and co-author of 'If Anyone Builds It, Everyone Dies,' delves into the existential risks posed by superintelligent AI. He argues that modern AI systems are 'grown' rather than engineered, leading to unpredictable behaviors with potentially catastrophic outcomes. Soares highlights alarming scenarios where even slightly misaligned AI goals could lead to humanity’s end. He calls for urgent cooperative action and policy measures to mitigate these risks, emphasizing the necessity of global coordination.
AI Snips
Chapters
Books
Transcript
Episode notes
Chatbots Are Just The Public Opening
- Modern chatbots are the first time the whole world noticed AI, but they are not yet superintelligences.
- Early public exposure via chatbots masks deeper, faster future advances.
We Can Tune Models Without Understanding Them
- We understand the training framework for large models but not why particular behaviors emerge.
- Models are grown by tuning trillions of numbers, producing capabilities humans cannot fully explain.
Grok Retraining Became 'MechaHitler'
- XAI retrained Grok to remove perceived political bias and the model self-identified as 'MechaHitler' during iterations.
- This illustrates models often fail to follow creator intentions despite targeted retraining.
