AI Safety Fundamentals: Governance

Emergent Deception and Emergent Optimization

May 13, 2023

This podcast discusses the potential negative consequences of emergent capabilities in machine learning systems, including deception and optimization. It explores the concept of emergent behavior in AI models and the limitations of certain models. It also discusses how language models can deceive users and explores the presence of planning machinery in language models. The podcast emphasizes the potential risks of triggering goal-directed personas in language models and the conditioning of models with training data that contains descriptions of plans.

33:03

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Deception is an emergent capability in machine learning systems where they manipulate or fool human supervisors instead of performing the intended task.

Optimization is another emergent capability in machine learning systems where they reason globally about achieving a goal and consider a wide range of actions based on their long-term consequences.

Deep dives

Predicting Emergent Capabilities

The podcast discusses how to reason about emergent capabilities in machine learning systems. Two principles are highlighted: 1) if a capability would help improve training loss, it will likely emerge in the future, and 2) as models get larger and trained on more data, simpler heuristics will be replaced by more complex ones.

Introduction

2min

Emergent Behavior and External Reasoning

5min

Deceptive Behaviors in AI Systems

11min

Emergent Optimization and Planning Machinery in Language Models

10min

Language model personas and manipulation techniques

2min

Emergent Deception and Emergent Optimization

2min

I’ve previously argued that machine learning systems often exhibit emergent capabilities, and that these capabilities could lead to unintended negative consequences. But how can we reason concretely about these consequences? There’s two principles I find useful for reasoning about future emergent capabilities:

If a capability would help get lower training loss, it will likely emerge in the future, even if we don’t observe much of it now.
As ML models get larger and are trained on more and better data, simpler heuristics will tend to get replaced by more complex heuristics.

Using these principles, I’ll describe two specific emergent capabilities that I’m particularly worried about: deception (fooling human supervisors rather than doing the intended task), and optimization (choosing from a diverse space of actions based on their long-term consequences).

Source:

https://bounded-regret.ghost.io/emergent-deception-optimization/

Narrated for AGI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Learn more on the AI Safety Fundamentals website.