The Challenges of Common Sensitivity in Pre-Training

It seems to open up very exciting research directions because it seems like a big open question. Then how do you reformulate the pre-training such that it's truly seeking, looking for correctness? I love the challenge of being able to abstract away this unspoken common sense knowledge about how the world works based on just raw internet data or YouTube videos.