AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
How to Pre-Trained Large Language Models
The thought was that you'd be pulling in these snippets that were taken out of context and not complete like the kind of data the LLM was trained on. So again like to me a lot of these projects are also about trying to uncover interesting properties of these various ways we can pre-trained large models from lots of data. And here it was kind of interesting to see that it had some amount of robustness, more than robustness it's actually can if you put in better information it would actually leverage it.