Improving Models with Expansion of Transformer Blocks

By introducing the expansion of transformer blocks, researchers have successfully enhanced existing models like Lama Pro 7b to Lama Pro 8.3b, focusing on improving programming and mathematics capabilities. The primary aim is to address catastrophic forgetting in neural networks, where specialized knowledge acquisition leads to sacrificing general knowledge. To tackle this issue, the researchers proposed creating new transformer blocks that are added on top of the existing model without altering the original learned knowledge. By training these additional blocks on specific tasks, such as coding, they demonstrated that Lama Pro can excel in both coding ability and general language proficiency, without compromising either. This breakthrough approach allows for a more effective model training process, showcases significant improvements over traditional models, and optimizes the trade-off between specialized and general knowledge in neural networks.