Federico Barbero, a lead author at DeepMind/Oxford, dives into the quirks of transformers and why large language models falter at tasks like counting. He reveals fascinating architectural bottlenecks that affect their performance. By drawing parallels with graph neural networks, he sheds light on the softmax function's role in limiting decision-making clarity. But not all hope is lost! Federico shares innovative 'glasses' to enhance transformer performance, including input tweaks and structural modifications to boost their clarity and efficiency.