AI Safety Fundamentals: Alignment

Embedded Agents

May 13, 2023

Computer scientist Scott Garrabrant discusses the challenge of building learning agents for real-world goals. The podcast explores the concept of embedded agents, the four complications of embedded agency, and open problems in world models and subsystem alignment. It also delves into the conflicts that arise when spinning up sub-agents with different goals.

17:39

Episode guests

Scott Garrabrant

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Embedded agents face challenges in optimizing realistic goals in physical environments without clear input-output channels.

Decision theory, embedded world models, robust delegation, and subsystem alignment are active areas of research to address these challenges.

Deep dives

Decision Theory

Decision theory explores the challenges of optimization for embedded agents. Dualistic models like argmax, which find actions that maximize rewards, do not apply well to agents embedded in environments without clear input-output channels. Major open problems in decision theory include reasoning about counterfactuals, handling multiple agent copies within an environment, and addressing logical uncertainty.

Introduction

2min

Embedded Agents: Challenges and Complications

9min

The Four Complications of Embedded Agency: Decision Theory and Embedded World Models

3min

Open problems in embedded world models, robust delegation, and subsystem alignment

2min

Spinning up Sub-Agents and the Challenge of Alignment

2min

Suppose you want to build a robot to achieve some real-world goal for you—a goal that requires the robot to learn for itself and figure out a lot of things that you don’t already know. There’s a complicated engineering problem here. But there’s also a problem of figuring out what it even means to build a learning agent like that. What is it to optimize realistic goals in physical environments? In broad terms, how does it work? In this series of posts, I’ll point to four ways we don’t currently know how it works, and four areas of active research aimed at figuring it out. This is Alexei, and Alexei is playing a video game. Like most games, this game has clear input and output channels. Alexei only observes the game through the computer screen, and only manipulates the game through the controller. The game can be thought of as a function which takes in a sequence of button presses and outputs a sequence of pixels on the screen. Alexei is also very smart, and capable of holding the entire video game inside his mind.

Original text:

https://intelligence.org/2018/10/29/embedded-agents/

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Learn more on the AI Safety Fundamentals website.