AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Back Prop information density: GPT3 = 1mm of Brain region
Back propagation is a more effective algorithm for squeezing knowledge into connections compared to the tool algorithm. Larger networks work better with forward propagation, while back propagation is not good at squeezing a lot of information into a few connections unless forced to. Large language models like GPT3 have trillions of connections, which is about the size of a cubic centimeter of cortex. These models know more facts than humans because they have read a lot of information on the web. However, the effectiveness of back propagation lies in how effectively it can get information into synapses and make use of experience.