AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
The Role of Algorithmic Progress in Scaling the Big Blob of Compute
The big blob of compute document, which I still have not made public, I probably should for like historical reasons. It might give us some sense of the kinds of things that matter and what don't. One was symmetries, which is basically like, if your architecture doesn't take into account the right kinds of symmetry, it doesn't work. And so again, things need to flow freely if they don't, it doesn’t work. This is why like Adam works better than normal SDD.