141 - Building an open source LM, with Iz Beltagy and Dirk Groeneveld
NLP Highlights
00:00
The Standard Recipe for Building Language Models
The real limitations we have or constraints that we have are a number of two few hours that we have for training. Our schedule, when we want to deliver or finish training the model and the available human human resources and engineering guide resource training. So we decided given all of these constraints to focus more on the data side and go with a fairly standard model. It's very possible that it is better to put the instruction data inside the part of the teaching data instead of a separate fine-tuning phase. We hope that when we make these language models open source, this encourages the research community more to study this recipe and see if there are better ways to do this.
Transcript
Play full episode
