Why Unsupervised Learning Isn't Working for RL?

BERT is very hard to get to work for reinforcement learning, where you pre-trained a very large model and then you fine-tune to any new task. It's not clear exactly what you should pre-train. On the other hand, something like GPTs are cool, you don't have to fine-t tune if you have a great model,. zero shot or a few shot will work at this time. So that is the advantage of training these large language models. You can ask if you had a choice, if you had 100 GPU, 1000 GPUs, would you go for training a GPT or an BERT? And the answer is you'd go for training