The Differences Between CNNs and Vision Transformers

Google Brain uncovers representation structure differences between CNNs and vision transformers. Andrey Kuznetsov: I found this pretty interesting. Some of the results are not very surprising, as you said. So for instance, in the early layers, closer to input, there's more global information and then resnets,. which makes sense due to the design transformers.