The Benefits of Information Retrieval and Model Internalization in Language Models

A model can store a large amount of information in its weights, including tactical knowledge, which allows for flexible behavior and making connections/nRetrieval has advantages such as access to real-time information, more detail, and human checkability/nModels should provide citations to assist with accurate supervision and verifiability of the output/nLarge language models can remember specific information from just one pass over the data/nThe model's ability to recall facts depends on how many times it has seen them, with an estimated range in the 10s