AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
The Importance of Context in Instruction Tuning
There's a relationship between train from scratch versus fine tune. A lot of times you want this large context, so you can stick in a lot of context that isn't in the base model. But there's other things you'd want a giant context window for which is just new information. I don't see a way around having a very large context window for that sort of thing.