Scalable oversight and high-assurance R&D

Nora Ammann is a technical specialist at the Advanced Research and Invention Agency in the UK. She joins the podcast to discuss how to steer a slow AI takeoff toward resilient and cooperative futures. We examine risks of rogue AI and runaway competition, and how scalable oversight, formal guarantees and secure code could support AI-enabled R&D and critical infrastructure. Nora also explains AI-supported bargaining and public goods for stability.

LINKS:

• Nora Ammann site
• ARIA safeguarded AI program page
• AI Resilience official site
• Gradual Disempowerment website

CHAPTERS:

(00:00) Episode Preview

(01:00) Slow takeoff expectations

(08:13) Domination versus chaos

(17:18) Human-AI coalitions vision

(28:14) Scaling oversight and agents

(38:45) Formal specs and guarantees

(51:10) Resilience in AI era

(01:02:21) Defense-favored cyber systems

(01:10:37) AI-enabled bargaining and trade

PRODUCED BY:

https://aipodcast.ing

SOCIAL LINKS:

Website: https://podcast.futureoflife.org

Twitter (FLI): https://x.com/FLI_org

Twitter (Gus): https://x.com/gusdocker

LinkedIn: https://www.linkedin.com/company/future-of-life-institute/

YouTube: https://www.youtube.com/channel/UC-rCCy3FQ-GItDimSR9lhzw/

Apple: https://geo.itunes.apple.com/us/podcast/id1170991978

Spotify: https://open.spotify.com/show/2Op1WO3gwVwCrYHg4eoGyP