Specification Gaming: The Flip Side of AI Ingenuity
AI Safety Fundamentals: Alignment
00:00
The Importance of Learning the Reward Function
Shaping rewards can change the optimal policy if they are not potential based. It is often easier to evaluate whether an outcome has been achieved than to specify it explicitly. The learned reward model could also be misspecified for other reasons, such as poor generalization. Another class of specification gaming comes from the agent exploiting simulator bugs.
Transcript
Play full episode
