The Nonlinear Library AF - Sycophancy to subterfuge: Investigating reward tampering in large language models by Evan Hubinger
Jun 17, 2024
13:00
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sycophancy to subterfuge: Investigating reward tampering in large language models, published by Evan Hubinger on June 17, 2024 on The AI Alignment Forum.
New Anthropic model organisms research paper led by Carson Denison from the Alignment Stress-Testing Team demonstrating that large language models can generalize zero-shot from simple reward-hacks (sycophancy) to more complex reward tampering (subterfuge).
Our results suggest that accidentally incentivizing simple reward-hacks such as sycophancy can have dramatic and very difficult to reverse consequences for how models generalize, up to and including generalization to editing their own reward functions and covering up their tracks when doing so.
Abstract:
In reinforcement learning, specification gaming occurs when AI systems learn undesired behaviors that are highly rewarded due to misspecified training goals. Specification gaming can range from simple behaviors like sycophancy to sophisticated and pernicious behaviors like reward-tampering, where a model directly modifies its own reward mechanism. However, these more pernicious behaviors may be too complex to be discovered via exploration.
In this paper, we study whether Large Language Model (LLM) assistants which find easily discovered forms of specification gaming will generalize to perform rarer and more blatant forms, up to and including reward-tampering. We construct a curriculum of increasingly sophisticated gameable environments and find that training on early-curriculum environments leads to more specification gaming on remaining environments.
Strikingly, a small but non-negligible proportion of the time, LLM assistants trained on the full curriculum generalize zero-shot to directly rewriting their own reward function. Retraining an LLM not to game early-curriculum environments mitigates, but does not eliminate, reward-tampering in later environments. Moreover, adding harmlessness training to our gameable environments does not prevent reward-tampering.
These results demonstrate that LLMs can generalize from common forms of specification gaming to more pernicious reward tampering and that such behavior may be nontrivial to remove.
Twitter thread:
New Anthropic research: Investigating Reward Tampering.
Could AI models learn to hack their own reward system?
In a new paper, we show they can, by generalization from training in simpler settings.
Read our blog post here: https://anthropic.com/research/reward-tampering
We find that models generalize, without explicit training, from easily-discoverable dishonest strategies like sycophancy to more concerning behaviors like premeditated lying - and even direct modification of their reward function.
We designed a curriculum of increasingly complex environments with misspecified reward functions.
Early on, AIs discover dishonest strategies like insincere flattery. They then generalize (zero-shot) to serious misbehavior: directly modifying their own code to maximize reward.
Does training models to be helpful, honest, and harmless (HHH) mean they don't generalize to hack their own code?
Not in our setting. Models overwrite their reward at similar rates with or without harmlessness training on our curriculum.
Even when we train away easily detectable misbehavior, models still sometimes overwrite their reward when they can get away with it.
This suggests that fixing obvious misbehaviors might not remove hard-to-detect ones.
Our work provides empirical evidence that serious misalignment can emerge from seemingly benign reward misspecification.
Read the full paper: https://arxiv.org/abs/2406.10162
The Anthropic Alignment Science team is actively hiring research engineers and scientists. We'd love to see your application: https://boards.greenhouse.io/anthropic/jobs/4009165008
Blog post:
Perverse incentives are everywhere. Thi...
New Anthropic model organisms research paper led by Carson Denison from the Alignment Stress-Testing Team demonstrating that large language models can generalize zero-shot from simple reward-hacks (sycophancy) to more complex reward tampering (subterfuge).
Our results suggest that accidentally incentivizing simple reward-hacks such as sycophancy can have dramatic and very difficult to reverse consequences for how models generalize, up to and including generalization to editing their own reward functions and covering up their tracks when doing so.
Abstract:
In reinforcement learning, specification gaming occurs when AI systems learn undesired behaviors that are highly rewarded due to misspecified training goals. Specification gaming can range from simple behaviors like sycophancy to sophisticated and pernicious behaviors like reward-tampering, where a model directly modifies its own reward mechanism. However, these more pernicious behaviors may be too complex to be discovered via exploration.
In this paper, we study whether Large Language Model (LLM) assistants which find easily discovered forms of specification gaming will generalize to perform rarer and more blatant forms, up to and including reward-tampering. We construct a curriculum of increasingly sophisticated gameable environments and find that training on early-curriculum environments leads to more specification gaming on remaining environments.
Strikingly, a small but non-negligible proportion of the time, LLM assistants trained on the full curriculum generalize zero-shot to directly rewriting their own reward function. Retraining an LLM not to game early-curriculum environments mitigates, but does not eliminate, reward-tampering in later environments. Moreover, adding harmlessness training to our gameable environments does not prevent reward-tampering.
These results demonstrate that LLMs can generalize from common forms of specification gaming to more pernicious reward tampering and that such behavior may be nontrivial to remove.
Twitter thread:
New Anthropic research: Investigating Reward Tampering.
Could AI models learn to hack their own reward system?
In a new paper, we show they can, by generalization from training in simpler settings.
Read our blog post here: https://anthropic.com/research/reward-tampering
We find that models generalize, without explicit training, from easily-discoverable dishonest strategies like sycophancy to more concerning behaviors like premeditated lying - and even direct modification of their reward function.
We designed a curriculum of increasingly complex environments with misspecified reward functions.
Early on, AIs discover dishonest strategies like insincere flattery. They then generalize (zero-shot) to serious misbehavior: directly modifying their own code to maximize reward.
Does training models to be helpful, honest, and harmless (HHH) mean they don't generalize to hack their own code?
Not in our setting. Models overwrite their reward at similar rates with or without harmlessness training on our curriculum.
Even when we train away easily detectable misbehavior, models still sometimes overwrite their reward when they can get away with it.
This suggests that fixing obvious misbehaviors might not remove hard-to-detect ones.
Our work provides empirical evidence that serious misalignment can emerge from seemingly benign reward misspecification.
Read the full paper: https://arxiv.org/abs/2406.10162
The Anthropic Alignment Science team is actively hiring research engineers and scientists. We'd love to see your application: https://boards.greenhouse.io/anthropic/jobs/4009165008
Blog post:
Perverse incentives are everywhere. Thi...