The Importance of Training Embedding Models

There seems to be a saturation point where you don't really get better embeddings after getting beyond some specific model capacity. Genentive on the other hand keeps on going better and better for as large as I've seen. We'll see how well this ages. And there have been some massive innovations in this space for the last four months. This has been something I've been following really closely. Maybe these types of things may become less relevant as attention becomes more efficient, he says.