How Do We Train Normal Language Models?

I think it's unclear what you should hope for in less clear cut cases and so for simplicity we don't focus on that case. I mostly mean it in the second sense so specifically how do we train normal language models um let's just say pre-trained language models like GPT 3 or whatever. We just have it basically predict the next token for a bunch of text on the internet and then you can prompt it in this way to get some answers from it which are usually pretty good if the model is big enough.

Transcript

Play full episode

Transcript

Episode notes

The AI-powered Podcast Player

Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!

Get the app