Future of Life Institute Podcast How to Avoid Two AI Catastrophes: Domination and Chaos (with Nora Ammann)
28 snips
Jan 7, 2026 Nora Ammann, a technical specialist at the UK’s ARIA focusing on AI safety, discusses crucial strategies for mitigating AI risks. She highlights the dangers of rogue AI dominance and chaotic competition, emphasizing the need for early interventions. Nora proposes human-AI coalitions to foster cooperative developments and scalable oversight. She explores the significance of using formal guarantees to enhance AI resilience and safety. Additionally, she examines the complexities of agent collaboration and the role of AI in improving cybersecurity.
AI Snips
Chapters
Books
Transcript
Episode notes
Slow Takeoff, Multiple Inflection Points
- Nora Ammann expects multiple capability inflection points rather than one sudden jump in AI progress.
- She calls this a slow takeoff and sees the next 2–4 years as path-defining for steering AI direction.
Two Distinct AI Failure Modes
- Two main failure clusters are domination by a powerful actor and diffuse chaos from excessive competition.
- Both threaten values: domination concentrates power; chaos erodes cooperation and public goods.
Build Scalable Oversight Tooling
- Build oversight, workflows and tooling so humans can trust AI outputs without blindly trusting systems.
- Aim for scalable methods that yield justified confidence in AI-produced science and engineering.
