Neural Networks Are Biased to Smooth Solutions

Why do tree-based models still outperform deep learning on tabular data? This question, in fact, can be asked again or reframed as which features of tabular data make this type of data easy to learn with tree-based methods and yet more difficult with a neural network. And it's probably one of the most interesting questions that I've been reading about in the last few years. So there are several findings. Two in particular are, in my opinion, very, very important and worth mention. Neural networks are biased to smooth solutions.