The Nonlinear Library AF - How We Picture Bayesian Agents by johnswentworth
Apr 8, 2024
11:19
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How We Picture Bayesian Agents, published by johnswentworth on April 8, 2024 on The AI Alignment Forum.
I think that when most people picture a Bayesian agent, they imagine a system which:
Enumerates every possible state/trajectory of "the world", and assigns a probability to each.
When new observations come in, loops over every state/trajectory, checks the probability of the observations conditional on each, and then updates via Bayes rule.
To select actions, computes the utility which each action will yield under each state/trajectory, then averages over state/trajectory weighted by probability, and picks the action with the largest weighted-average utility.
Typically, we define Bayesian agents as agents which behaviorally match that picture.
But that's not really the picture David and I typically have in mind, when we picture Bayesian agents. Yes, behaviorally they act that way. But I think people get overly-anchored imagining the internals of the agent that way, and then mistakenly imagine that a Bayesian model of agency is incompatible with various features of real-world agents (e.g. humans) which a Bayesian framework can in fact handle quite well.
So this post is about our prototypical mental picture of a "Bayesian agent", and how it diverges from the basic behavioral picture.
Causal Models and Submodels
Probably you've heard of
causal diagrams or Bayes nets by now.
If our Bayesian agent's world model is represented via a big causal diagram, then that already looks quite different from the original "enumerate all states/trajectories" picture. Assuming reasonable sparsity, the data structures representing the causal model (i.e. graph + conditional probabilities on each node) take up an amount of space which grows linearly with the size of the world, rather than exponentially. It's still
too big for an agent embedded in the world to store in its head directly, but much smaller than the brute-force version.
(Also, a realistic agent would want to explicitly represent more than just one causal diagram, in order to have uncertainty over causal structure. But that will largely be subsumed by our next point anyway.)
Much more efficiency can be achieved by
representing causal models like we represent programs. For instance, this little "program":
… is in fact a recursively-defined causal model. It compactly represents an infinite causal diagram, corresponding to the unrolled computation. (See the linked post for more details on how this works.)
Conceptually, this sort of representation involves lots of causal "submodels" which "call" each other - or, to put it differently, lots of little diagram-pieces which can be wired together and reused in the full world-model. Reuse means that such models can represent worlds which are "bigger than" the memory available to the agent itself, so long as those worlds have lots of compressible structure - e.g.
the factorial example above, which represents an infinite causal diagram using a finite representation.
(Aside: those familiar with probabilistic programming could view this world-model representation as simply a probabilistic program.)
Updates
So we have a style of model which can compactly represent quite large worlds, so long as those worlds have lots of compressible structure. But there's still the problem of updates on that structure.
Here, we typically imagine some kind of message-passing, though it's an open problem exactly what such an algorithm looks like for big/complex models.
The key idea here is that most observations are not directly relevant to our submodels of most of the world. I see a bird flying by my office, and that tells me nothing at all about the price of gasoline[1]. So we expect that, the vast majority of the time, message-passing updates of a similar flavor to those used on Bayes nets (though not exactl...
I think that when most people picture a Bayesian agent, they imagine a system which:
Enumerates every possible state/trajectory of "the world", and assigns a probability to each.
When new observations come in, loops over every state/trajectory, checks the probability of the observations conditional on each, and then updates via Bayes rule.
To select actions, computes the utility which each action will yield under each state/trajectory, then averages over state/trajectory weighted by probability, and picks the action with the largest weighted-average utility.
Typically, we define Bayesian agents as agents which behaviorally match that picture.
But that's not really the picture David and I typically have in mind, when we picture Bayesian agents. Yes, behaviorally they act that way. But I think people get overly-anchored imagining the internals of the agent that way, and then mistakenly imagine that a Bayesian model of agency is incompatible with various features of real-world agents (e.g. humans) which a Bayesian framework can in fact handle quite well.
So this post is about our prototypical mental picture of a "Bayesian agent", and how it diverges from the basic behavioral picture.
Causal Models and Submodels
Probably you've heard of
causal diagrams or Bayes nets by now.
If our Bayesian agent's world model is represented via a big causal diagram, then that already looks quite different from the original "enumerate all states/trajectories" picture. Assuming reasonable sparsity, the data structures representing the causal model (i.e. graph + conditional probabilities on each node) take up an amount of space which grows linearly with the size of the world, rather than exponentially. It's still
too big for an agent embedded in the world to store in its head directly, but much smaller than the brute-force version.
(Also, a realistic agent would want to explicitly represent more than just one causal diagram, in order to have uncertainty over causal structure. But that will largely be subsumed by our next point anyway.)
Much more efficiency can be achieved by
representing causal models like we represent programs. For instance, this little "program":
… is in fact a recursively-defined causal model. It compactly represents an infinite causal diagram, corresponding to the unrolled computation. (See the linked post for more details on how this works.)
Conceptually, this sort of representation involves lots of causal "submodels" which "call" each other - or, to put it differently, lots of little diagram-pieces which can be wired together and reused in the full world-model. Reuse means that such models can represent worlds which are "bigger than" the memory available to the agent itself, so long as those worlds have lots of compressible structure - e.g.
the factorial example above, which represents an infinite causal diagram using a finite representation.
(Aside: those familiar with probabilistic programming could view this world-model representation as simply a probabilistic program.)
Updates
So we have a style of model which can compactly represent quite large worlds, so long as those worlds have lots of compressible structure. But there's still the problem of updates on that structure.
Here, we typically imagine some kind of message-passing, though it's an open problem exactly what such an algorithm looks like for big/complex models.
The key idea here is that most observations are not directly relevant to our submodels of most of the world. I see a bird flying by my office, and that tells me nothing at all about the price of gasoline[1]. So we expect that, the vast majority of the time, message-passing updates of a similar flavor to those used on Bayes nets (though not exactl...