#66 – Michael Cohen on Input Tampering in Advanced RL Agents
Hear This Idea
00:00
The Importance of Observation in a Chatbot
A Mario agent that learns to attend to the score at the bottom of the screen would likely try to intervene in its observation of the score. And what's critical is that point has to be unobservable to it, the score,. Because as it's continuing to act, it can continue to get information that it treats as goal information. In my terminology, this is reward maximization with a huge inductive bias in favor of reward just being about hearing certain words from your human interlocutor. That could start to feel like the language I'm using is a bit awkward.
Play episode from 01:33:06
Transcript
