03: The Next Generation of LLMs with Jonathan Frankle of MosaicML
Replit AI Podcast
00:00
How to Evaluate a Generative Model
If you're going to train a 30B model, you may as well do it in 8K. Is there any accuracy cost? It's an interesting question. We haven't seen any so far. The really popular data set C4 has basically nothing that's longer than 4096. That is also the data set that tends to do best on multiple choice tasks.
Transcript
Play full episode
