The difference lies between task-specific models and encoder models used as foundations for task-specific models. Encoder models like GPT-3 are trained with a language modeling objective and serve as pre-trained weights for task-specific models. Transfer learning often involves training a task-specific network on top of these pre-trained weights, leveraging the knowledge about language and the world encoded in them.