The Importance of Grounded Language Learning in Machine Translation

The whole like module net project came about halfway through my PhD. I was trying to figure out if you could do instruction following without logical forms by just saying, I have my sentence and maybe I have some structured representation of the sentence in the form of a dependency parse. And then that went reasonably well and next thing that I wanted to do was visual question answering. We didn't at the time have good tools for answering questions about things like images or videos or other unstructured representations of knowledge. It's nice to have these sort of abstract logical representations of sentence meaning say, but for lots of tasks we actually want to use language to interact with computer systems.