The Challenges of Measurement in Language Models

The amount of data seems like it's a real ball. Right now also the quality of the data seems, seems limited with that shotgun approach. If you take the internet and instead of training your large language model on internet text, you, you partial it up in small pieces and then train it, then you never get long context windows. But if you shot get approached, you're not going to get any of that.