The Inductive Bias and the Bottleneck in Deep Learning

There's a curse of dimensionality. We cannot predict one pixel given just two or three or four others. You cannot tell a story about pixels except if you bring it up to the high level where we have stories. And so at the high level, we have these variables that satisfy these sparsity and bottleneck constraints. But not everything can be expressed in words. That's where deep learning and its current form is beating traditional programming. Because we are able to learn complicated objects, distributions that cannot be reduced to language like things.