Symbols Help Distinguish Information

Once you apply a hard attention, you're essentially discarded so much information that you could have passed on. Yet somehow it just seems like fundamentally it's less efficient to learn that way. Well, okay. So inducted bias is like regularizers in machinery. They make your life harder. In a sense, that restricts your capacity.