Using Pre-Trained Self-Supervised Representations in Deep Learning?

It's difficult to not take advantage of self-supervised learning anywhere right now in deep learning. So just like that, I'm also leveraging that for my current research. CLIP you can consider the system's kind of like giant self-super supervised contrast learning done on like internet-scale data.