How to Evaluate a Generative Model

If you're going to train a 30B model, you may as well do it in 8K. Is there any accuracy cost? It's an interesting question. We haven't seen any so far. The really popular data set C4 has basically nothing that's longer than 4096. That is also the data set that tends to do best on multiple choice tasks.