AI Safety Fundamentals: Alignment

Cooperation, Conflict, and Transformative Artificial Intelligence: Sections 1 & 2 — Introduction, Strategy and Governance

May 13, 2023

Jesse Clifton, a researcher in transformative artificial intelligence, discusses the potential consequences and risks of TAI. They explore cooperation failures, social dilemmas, and the strategic landscape and governance of TAI systems. Topics include bargaining models of war, simulated conflict, and the importance of credible commitments in multi-agent systems.

27:32

Episode guests

Jesse Clifton

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Preventing catastrophic failures of cooperation among TAI systems is crucial to ensure safe and beneficial development of transformative artificial intelligence.

The research agenda in this field should include studying AI misalignment scenarios, analyzing the impact of formal research on real-world decisions, and considering the potential downsides of increased technical understanding and idealized models of rationality.

Deep dives

Main ideas and insights

The podcast discusses the research agenda on Cooperation, Conflict, and Transformative Artificial Intelligence (TAI). It emphasizes the importance of preventing catastrophic failures of cooperation among TAI systems. The podcast explores the concept of cooperation failures and their potential consequences, including destructive conflict, coercion, and social dilemmas. It highlights the need to study the strategic landscape, AI alignment scenarios, commitment and transparency, and potential downsides of research on cooperation failures.

Introduction

3min

Cooperation Failure and Social Dilemma

4min

Social Dilemmas, Bargaining Models of War, and Simulated Conflict

5min

Understanding the Strategic Landscape and Governance of Transformative Artificial Intelligence (TAI) Systems

5min

Credibility and Commitments in Multi-Agent Systems

9min

Transformative artificial intelligence (TAI) may be a key factor in the long-run trajectory of civilization. A growing interdisciplinary community has begun to study how the development of TAI can be made safe and beneficial to sentient life (Bostrom 2014; Russell et al., 2015; OpenAI, 2018; Ortega and Maini, 2018; Dafoe, 2018). We present a research agenda for advancing a critical component of this effort: preventing catastrophic failures of cooperation among TAI systems. By cooperation failures we refer to a broad class of potentially-catastrophic inefficiencies in interactions among TAI-enabled actors. These include destructive conflict; coercion; and social dilemmas (Kollock, 1998; Macy and Flache, 2002) which destroy value over extended periods of time. We introduce cooperation failures at greater length in Section 1.1. Karnofsky (2016) defines TAI as ''AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution''. Such systems range from the unified, agent-like systems which are the focus of, e.g., Yudkowsky (2013) and Bostrom (2014), to the "comprehensive AI services’’ envisioned by Drexler (2019), in which humans are assisted by an array of powerful domain-specific AI tools. In our view, the potential consequences of such technology are enough to motivate research into mitigating risks today, despite considerable uncertainty about the timeline to TAI (Grace et al., 2018) and nature of TAI development.

Original text:

https://www.alignmentforum.org/s/p947tK8CoBbdpPtyK/p/KMocAf9jnAKc2jXri

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Learn more on the AI Safety Fundamentals website.