6min chapter

Machine Learning Street Talk (MLST) cover image

#55 Self-Supervised Vision Models (Dr. Ishan Misra - FAIR).

Machine Learning Street Talk (MLST)

CHAPTER

Is There a Difference Between Self-Supervised Learning and DNO?

When I started working at Facebook on self-supervised learning, we started using ResNet because well, it was the architecture that actually made more sense and we could have further comparisons to supervised. So basically if you look at even standard supervised learning like efficient nets and so on, they use they train the model but they maintain the sort of momentum encoder. It's not really used but at the end when they're evaluating on image net or something they use to momentum encoder because they find that that does better. This is like the standard pull yack averaging. Having this pull yack averaged actually helps you because all the sort of random steps you take in the feature space get averaged out

00:00

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.
App store bannerPlay store banner

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode