The High Level Takeaway for Efficiency Methods

The key question is whether the knowledge that a large model has should be stored in its parameters. And I think we've seen some promising results with models that are order of magnitude smaller than say, GPT-3. We're still using the same transformer architecture for 2016 or 2017,. which is pretty remarkable if you think about it.