It's not just that you are trying to understand the relationships between things, but what you need to make relationships or discover relationships over. I think it needs more degrees of freedom and the current models have. And when you want to deal with non-diverrunchable ones, you need to have an attention system that is discrete and though dimensional. It can still be in your network, but not a fully differentiable one.