How to Reduce the Model Capacity

The first thing that we would be looking into is finding a good, large model that would be performing the task well. And having that as a benchmark, we would be then looking into how to reduce the model size. But that might not necessarily come from the distillation first, it would be coming from maybe simplifying the architecture of the model. Usually they work together and I wouldn't say that I would prefer one particular technique over another.