The Challenge of Fine Tuning GPUs

The goal usually is for the fine tuning step, hopefully you can get by with CPU's even. It depends on how large your model is, I mean, whether it can fit in memory and how much operation they need to do. The major challenge of fine tuning is can we get the good few short results that we see with huge models on much smaller models? We're seeing a GPD-3 doing wonderfully with only a handful of examples but smaller models still need quite a bit of them.