The Importance of Pre-Training Language Models

Tom McCann, co-founder and CTO of nanohink.com has been working on NLP since his early days at Stanford University. He says he wanted to replicate the magic image net and convolutional neural networks in pre trained language sentences. And so as a first step, we wanted to pre train an encoder in the context. That paper had a very short moment in the sun before Elmo already made it even better.