Challenges in Mixing Instructions, Parameters, and Retrieved Documents in Language Models

This chapter explores the difficulties faced in combining instructions, parameters, and retrieved documents in language models (LMs). It emphasizes the need for designing an architecture that can effectively separate the control and data channels to ensure optimal use of retrieved models. The chapter also discusses the importance of detecting out-of-distribution queries, training LMs on a wide range of variability, uncovering missing variability for underrepresented sub-communities, and incorporating checkers and validators to ensure quality and safety in code generation and other structured objects. Additionally, the speakers highlight the significance of developing a prefrontal cortex module to accompany LLMs, using external tools like proof assistants, constraint checkers, sat solvers, and numerical codes for structural integrity checks, and integrating feedback from error checking into code generation offerings. The chapter concludes with a message to graduate students, urging them to pursue further research and innovative solutions beyond LLMs.