The Complexity of CPUs

An error in the central equation of AI, known as the attention formula, is causing difficulties in compressing and deploying transformer models. The author proposes a simple tweak to adding one to the denominator of the softmax function that could potentially solve this crippling issue. There's a debate around whether this would make a significant difference with one of the main points being the network's existing ability to opt out by providing a near-zero output vector.

Transcript

Play full episode

The AI-powered Podcast Player

Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!

Get the app