Transfer Learning for Text - Can You Get Natural Text Out of Common Crawl?

Pipe line could be used to get access to large volumes of text data that can be useful for projects like this. It's the really nice thing about transfer learning for text, all of the success comes from unsupervised pre training. We did a pretty good job of getting clean text out of common crawl but there are definitely additional things that could be done to make sure the text is as natural as possible.