AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Is There a Form of Compositional Representation in These Models?
There is emerging evidence from this line of work that shows that indeed, there might be a form of compositional representation in these models. That is not classical. It does not involve this discrete constituent structure where you can take the representation for dog, mine and biting the biting relation and literally just concatenate these into a complex representation. But it's doing a more sophisticated form of composition that can still keep certain attributes of the different words neatly separated by reading and writing information into distinct subspaces. So again, if you think of the idea that the model tracks the fact that one of the words is the verb, and that the verb relates to the subject in subsets, you might want