AXRP - the AI X-risk Research Podcast
25 - Cooperative AI with Caspar Oesterheld
Episode guests
Podcast summary created with Snipd AI
Quick takeaways
- Cooperative AI research can address conflicts and opportunities between AI systems developed by different parties.
- Regret minimization may not capture certain scenarios where randomization or specific actions can lead to better outcomes.
- Bounded inductive rationality provides a decision-theoretical foundation applicable to game theory settings.
- The auction-based decision-making framework offers a flexible approach for strategic decision-making in game theory.
- Safe Pareto improvements aim to address equilibrium selection and coordination problems in game theory.
- Clear communication, mutual understanding, and collaborative decision-making are crucial for effective implementation of safe Pareto improvements.
Deep dives
The Comparison Between Cooperative AI and X-Risks
This podcast episode explores the relevance of cooperative AI research to existential risks (X-Risks). It discusses how cooperative AI can address the potential conflicts and cooperation opportunities between AI systems developed by different parties, such as governments or companies. The episode highlights the role of AI in international relations and how disagreements between governments, even those involving nuclear weapons, could pose significant risks if AI systems are involved in decision-making. It also acknowledges the challenges of applying game theory and highlights the need for AI-specific approaches to train AI systems to make good strategic decisions.
Critique of Regret Minimization and the Need for a New Approach
The podcast episode critiques the limitations of regret minimization as a decision-making criterion, particularly in game theory settings. It discusses how regret minimization may fail to account for certain scenarios where randomization or specific actions may lead to better outcomes but are not captured in the regret minimization framework. The episode highlights the importance of considering the environment's response to the decision-maker's actions and the need for a new approach that better integrates decision theory, game theory, and the modeling of decision-makers by their environments.
An Alternative: The Theory of Bounded Inductive Rationality
The podcast episode introduces the theory of bounded inductive rationality as an alternative approach to decision-making in game theory settings. It explains that the theory involves running an auction between hypotheses, where each hypothesis offers a recommendation and an estimate of the expected utility. The decision-maker follows the recommendation of the highest bidder. The episode emphasizes that this approach takes into account the modeling of the decision-maker by the environment, allows for strategic reasoning, and considers randomization as a significant factor in decision-making. The theory aims to provide a decision-theoretical foundation that is bounded and applicable to game theory settings.
Auction-Based Decision-Making and the Benefits of the Framework
The podcast episode details the algorithm of the auction-based decision-making framework. It describes how hypotheses are given virtual currency and engage in an auction, bidding based on their recommendations and utility estimates. The winning hypothesis is selected, and the decision-maker follows their recommendation, with the hypothesis paying the bid amount. The episode emphasizes that the framework provides a flexible approach that allows for strategic decision-making, mitigates overconfidence or underconfidence in recommendations, and encourages hypotheses to provide accurate utility estimates. It offers a promising alternative to regret minimization, addressing some of its limitations and providing a more comprehensive decision-making approach for game theory settings.
Safe Pareto Improvements
The paper explores the concept of safe Pareto improvements in game theory. Safe Pareto improvements aim to address the issue of equilibrium selection and coordination problems in games, where multiple equilibria and conflicting preferences can lead to undesirable outcomes. The idea is to design a strategy or instruction that aligns the actions of players in ways that improve overall outcomes without making anyone worse off. This can involve adopting new attitudes, committing to specific actions, or modifying utility functions to ensure more favorable outcomes. The focus is on finding solutions that are safe, meaning they do not rely on assumptions about equilibrium selection or guesses about the opponent's moves. By implementing safe Pareto improvements, players can achieve better outcomes and mitigate the challenges posed by equilibrium selection problems in game theory.
Applicability in Real-Life Situations
While the concept of safe Pareto improvements provides a promising approach to address coordination and equilibrium selection problems, its applicability in real-life situations may require more complex implementation. In practice, it may involve delegating decisions to AI systems or experts and providing meta-instructions for strategic scenarios. Importantly, the instructions should ensure better outcomes for all players involved, not just the delegating party. This helps avoid competitive dynamics and promotes collaborative solutions. However, the implementation and design of safe Pareto improvements in real-life scenarios may be more challenging due to the complexity of interactions, multiple equilibria, and conflicting preferences. Further research and analysis are needed to explore the practical implementation and effectiveness of safe Pareto improvements.
Challenges and Considerations
There are several challenges and considerations in utilizing safe Pareto improvements. The existence of multiple potential safe Pareto improvements may require coordination between players to agree on which improvement to adopt. This can be especially complex when preferences or equilibria are uncertain or when players have conflicting views. Additionally, the design of meta-instructions or strategies for AI systems needs careful consideration to ensure they align with the desired outcomes. While safe Pareto improvements offer a valuable framework for improving outcomes in coordination problems, they also highlight the need for clear communication, mutual understanding, and collaborative decision-making processes among players.
Summary - Bounded Rational Inductive Agents
The paper introduces bounded rational inductive agents that can cooperate in the Prisoner's Dilemma against copies of themselves. The agents reason based on a surrogate goal, which is a hypothetical goal that represents the original goal of cooperation. By adopting this approach, the agents can overcome difficulties such as learning to cooperate and avoiding defection when faced with copies. The paper presents the agent's decision-making process and demonstrates through experiments that the surrogate goal strategy can lead to increased cooperation.
Summary - Safe Baseline Improvements
The paper discusses the concept of safe baseline improvements in AI cooperation. Safe baseline improvements involve modifying AI policies to cooperate more when facing similar policies and defect more against dissimilar ones. The authors propose a framework called program equilibrium, where AI agents submit programs that receive similarity-based information about opponents and determine whether to cooperate or defect. The paper highlights the importance of finding cooperative equilibria against copies and explores the challenges and potential solutions for achieving such equilibria through alternating best-response training and pre-training methods.
Summary - Similarity-Based Cooperation
The paper explores similarity-based cooperation in game theory. It introduces a setting where two players submit policies that receive similarity signals about one another and make decisions accordingly. The aim is to cooperate with similar policies and defect against dissimilar ones. The authors present various similarity functions and examine the equilibria and payoffs in prisoner's dilemma games. The paper also discusses the limitations and potential of pre-training and opponent shaping methods in promoting cooperation.
Summary - Focal Lab at CMU
The Focal Lab at Carnegie Mellon University, led by Vincent Conitzer, focuses on foundational research in cooperative AI. The lab explores topics such as bounded rational inductive agents, safe baseline improvements, similarity-based cooperation, and decision theory. The lab seeks to develop theoretical frameworks and practical approaches for promoting cooperation and addressing challenges in multi-agent scenarios. Prospective PhD students are encouraged to consider working with the lab and can explore opportunities for research in these areas.
Imagine a world where there are many powerful AI systems, working at cross purposes. You could suppose that different governments use AIs to manage their militaries, or simply that many powerful AIs have their own wills. At any rate, it seems valuable for them to be able to cooperatively work together and minimize pointless conflict. How do we ensure that AIs behave this way - and what do we need to learn about how rational agents interact to make that more clear? In this episode, I'll be speaking with Caspar Oesterheld about some of his research on this very topic.
Patreon: patreon.com/axrpodcast
Ko-fi: ko-fi.com/axrpodcast
Episode art by Hamish Doodles: hamishdoodles.com
Topics we discuss, and timestamps:
- 0:00:34 - Cooperative AI
- 0:06:21 - Cooperative AI vs standard game theory
- 0:19:45 - Do we need cooperative AI if we get alignment?
- 0:29:29 - Cooperative AI and agent foundations
- 0:34:59 - A Theory of Bounded Inductive Rationality
- 0:50:05 - Why it matters
- 0:53:55 - How the theory works
- 1:01:38 - Relationship to logical inductors
- 1:15:56 - How fast does it converge?
- 1:19:46 - Non-myopic bounded rational inductive agents?
- 1:24:25 - Relationship to game theory
- 1:30:39 - Safe Pareto Improvements
- 1:30:39 - What they try to solve
- 1:36:15 - Alternative solutions
- 1:40:46 - How safe Pareto improvements work
- 1:51:19 - Will players fight over which safe Pareto improvement to adopt?
- 2:06:02 - Relationship to program equilibrium
- 2:11:25 - Do safe Pareto improvements break themselves?
- 2:15:52 - Similarity-based Cooperation
- 2:23:07 - Are similarity-based cooperators overly cliqueish?
- 2:27:12 - Sensitivity to noise
- 2:29:41 - Training neural nets to do similarity-based cooperation
- 2:50:25 - FOCAL, Caspar's research lab
- 2:52:52 - How the papers all relate
- 2:57:49 - Relationship to functional decision theory
- 2:59:45 - Following Caspar's research
The transcript: axrp.net/episode/2023/10/03/episode-25-cooperative-ai-caspar-oesterheld.html
Links for Caspar:
- FOCAL at CMU: www.cs.cmu.edu/~focal/
- Caspar on X, formerly known as Twitter: twitter.com/C_Oesterheld
- Caspar's blog: casparoesterheld.com/
- Caspar on Google Scholar: scholar.google.com/citations?user=xeEcRjkAAAAJ&hl=en&oi=ao
Research we discuss:
- A Theory of Bounded Inductive Rationality: arxiv.org/abs/2307.05068
- Safe Pareto improvements for delegated game playing: link.springer.com/article/10.1007/s10458-022-09574-6
- Similarity-based Cooperation: arxiv.org/abs/2211.14468
- Logical Induction: arxiv.org/abs/1609.03543
- Program Equilibrium: citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=e1a060cda74e0e3493d0d81901a5a796158c8410
- Formalizing Objections against Surrogate Goals: www.alignmentforum.org/posts/K4FrKRTrmyxrw5Dip/formalizing-objections-against-surrogate-goals
- Learning with Opponent-Learning Awareness: arxiv.org/abs/1709.04326
Remember Everything You Learn from Podcasts
