The approach described addresses the challenge of catastrophic forgetting by enabling the integration of new knowledge through the introduction of new experts rather than retraining existing ones. Managing a vast number of experts, the methodology emphasizes the necessity for efficient selection during both training and inference phases. This efficiency is achieved through a technique called product key retrieval, drawing inspiration from search engine algorithms to swiftly identify and deploy the relevant experts for specific tasks. This method highlights the importance of leveraging scaling laws and advanced retrieval techniques to enhance performance in dynamic learning environments.
Our 175th episode with a summary and discussion of last week's big AI news!
With hosts Andrey Kurenkov (https://twitter.com/andrey_kurenkov) and Jeremie Harris (https://twitter.com/jeremiecharris)
In this episode of Last Week in AI, hosts Andrey Kurenkov and Jeremy Harris explore recent AI advancements including OpenAI's release of GPT 4.0 Mini and Mistral’s open-source models, covering their impacts on affordability and performance. They delve into enterprise tools for compliance, text-to-video models like Hyper 1.5, and YouTube Music enhancements. The conversation further addresses AI research topics such as the benefits of numerous small expert models, novel benchmarking techniques, and advanced AI reasoning. Policy issues including U.S. export controls on AI technology to China and internal controversies at OpenAI are also discussed, alongside Elon Musk's supercomputer ambitions and OpenAI’s Prover-Verify Games initiative.
Read out our text newsletter and comment on the podcast at https://lastweekin.ai/
If you would like to become a sponsor for the newsletter, podcast, or both, please fill out this form.
Email us your questions and feedback at contact@lastweekinai.com and/or hello@gladstone.ai
Timestamps + links:
- (00:00:00) AI Song Intro
- (00:00:40) Intro / Banter
- Tools & Apps
- Applications & Business
- Projects & Open Source
- Research & Advancements
- Policy & Safety
- (01:44:59) Outro + AI Song