How to Predict a Token in Natural Language

In a paper, we tried to enumerate what some of these are for language modeling. We did this by clustering together samples based on whether the model's gradient was similar for those samples. And you can find just a large number of clusters where it's the same type of problem or doing prediction on those samples requires the same piece of knowledge. So anyway, it's complicated and weird. Who knows whether this is even the right model for thinking about the networks. But it would be very exciting if it was the true model because it would maybe tell you that there were these set of things where if you enumerated them, you could understand the network's performance.