Alex Tamkin on Self-Supervised Learning and Large Language Models
The Gradient: Perspectives on AI
00:00
The Benefits of ImageNet for Language Modeling
ImageNet has like a thousand classes and then that's about it, even though obviously there's way more. So you're inherently trying to kind of understand the associations and correlations and so on without human-labeled secondary task. And maybe that sort of raw-based sort of form of a task builds a whole lot of a broader range of capabilities than if you were to say, spend a whole lots of time and get humans to construct a label-based learning signal.
Transcript
Play full episode
