Convergence on a Small Set of Effective Models

In the field of machine learning, researchers have witnessed a convergence towards a select few models, particularly those in the Bert family such as Roberta Bert. These models, known as foundation models, have consistently performed well and are widely used within the community. Despite the variety of models previously introduced, the community now favors these specific models for their effectiveness. Moreover, it is possible to train these models using language modeling approaches without requiring excessive computational resources, making it accessible for specific datasets or languages.