By introducing the expansion of transformer blocks, researchers have successfully enhanced existing models like Lama Pro 7b to Lama Pro 8.3b, focusing on improving programming and mathematics capabilities. The primary aim is to address catastrophic forgetting in neural networks, where specialized knowledge acquisition leads to sacrificing general knowledge. To tackle this issue, the researchers proposed creating new transformer blocks that are added on top of the existing model without altering the original learned knowledge. By training these additional blocks on specific tasks, such as coding, they demonstrated that Lama Pro can excel in both coding ability and general language proficiency, without compromising either. This breakthrough approach allows for a more effective model training process, showcases significant improvements over traditional models, and optimizes the trade-off between specialized and general knowledge in neural networks.
Our 151st episode with a summary and discussion of last week's big AI news!
Check out our sponsor, the SuperDataScience podcast. You can listen to SDS across all major podcasting platforms (e.g., Spotify, Apple Podcasts, Google Podcasts) plus there’s a video version on YouTube.
Read out our text newsletter and comment on the podcast at https://lastweekin.ai/
Email us your questions and feedback at contact@lastweekin.ai
Timestamps + links:
- (00:00:00) Intro / Banter
- Tools & Apps
- Applications & Business
- Projects & Open Source
- Research & Advancements
- Policy & Safety
- Synthetic Media & Art
- (01:35:15) Outro