The Pros and Cons of Visual Model Based Learning for Generalist Robots

The key idea is that we want robots to be able to do lots of different tasks. And one approach for doing that is to essentially learn these fairly general purpose models which try to predict what the future will look like as a function of the actions the robot takes. Overall, I think that this approach is promising because it's self-supervised, because it allows you to solve a number of different tasks and so forth. But there are a number of challenges with it as well. It's hard to train video prediction models on very broad data sets; but I have reason to be optimistic about this.