Today we are going to talk about the *Data-efficient image Transformers paper or (DeiT) which Hugo is the primary author of. One of the recipes of success for vision models since the DL revolution began has been the availability of large training sets. CNNs have been optimized for almost a decade now, including through extensive architecture search which is prone to overfitting. Motivated by the success of transformers-based models
in Natural Language Processing there has been increasing attention in applying these approaches to vision models. Hugo and his collaborators used a different training strategy and a new distillation token to get a massive increase in sample efficiency with image transformers.
00:00:00 Introduction
00:06:33 Data augmentation is all you need
00:09:53 Now the image patches are the convolutions though?
00:12:16 Where are those inductive biases hiding?
00:15:46 Distillation token
00:21:01 Why different resolutions on training
00:24:14 How data efficient can we get?
00:26:47 Out of domain generalisation
00:28:22 Why are transformers data efficient at all? Learning invariances
00:32:04 Is data augmentation cheating?
00:33:25 Distillation strategies - matching the intermediatae teacher representation as well as output
00:35:49 Do ML models learn the same thing for a problem?
00:39:01 How is it like at Facebook AI?
00:41:17 How long is the PhD programme?
00:42:03 Other interests outside of transformers?
00:43:18 Transformers for Vision and Language
00:47:40 Could we improve transformers models? (Hybrid models)
00:49:03 Biggest challenges in AI?
00:50:52 How far can we go with data driven approach?
