Orca: Progressive Learning from Complex Explanation Traces of GPT-4
Deep Papers
00:00
The Challenges With GPT for Bias
One thing that was kind of interesting for me here was the limited imitation signals right so it's like you train these things on query response pairs from say a big model. But then they also talk about other like, at least theoretical ways you could get imitation signals like logits intermediate representation representations or states. And I'm not quite sure if and when there is some kind of open source version of GPT for some kind of mega model that's open source, you could do something like this. So I thought that was an interesting idea.
Transcript
Play full episode
