Scaling Multi-Modal Generative AI with Luke Zettlemoyer - #650
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
00:00
The Future of Instruction Back Translation
Shand did a great job on the paper, discussing the pre-training and fine-tuning stages of language models. The distinction between the two may become blurred over time. The idea is to use back translation to gather a large amount of training data. This involves having the model write the input based on desired output examples. By scaling up the training, better and more versatile models can be achieved. The approach can be generalized to different types of content, but covering various domains is important for optimal results. Additionally, fine-tuning can unlock specific abilities like poetry generation, while still requiring exposure to prompts for improved performance in specific areas.
Play episode from 16:50
Transcript
