Defining Multimodal Models and Their Training Challenges

The multimodal language modeling space is still evolving, with many researchers trying to determine the optimal use cases for multimodal models compared to traditional language-only models. Late fusion models, which integrate a language backbone with an image encoder, have gained popularity due to their stability and predictability, even though they may be costly to fine-tune. This approach has been adopted in recent models like Molmo and LAMA 3.2 Vision, with ongoing discussions about the potential benefits of early fusion models when tested on larger datasets. Unresolved questions also surround how standard evaluation benchmarks, primarily designed for language models, may perform differently with multimodal training.