How Do We Learn Language Conditional Agents?

I feel like we are not quite there yet, even in the sense of how do we formulate the problem? What is the kind of data we need to train the robert like that? Where do we get the language data? I think all these are latically unsolved questions. If anybody can make this wore, if anybody knows what's missing from clip to make this work, at the villy, i imagine people will figure it out in the next couple of years,. but i don't know when.