Last Week in AI #190 - AI scaling struggles, OpenAI Agents, Super Weights
143 snips
Nov 28, 2024 This discussion dives into the recent scaling struggles faced by major AI companies like OpenAI and Google. A new AI agent tool from OpenAI aims to automate user tasks, while Google's Gemini model impressively tops the LLM leaderboard. The podcast also explores a hefty $100 billion proposal for an AI data center and the concept of 'Super Weights' in language models, revealing intriguing insights into their performance. Additionally, the conversation touches on safety shifts at OpenAI and the evolving landscape of AI infrastructure and investment.
AI Snips
Chapters
Transcript
Episode notes
Superweights in LLMs
- LLMs possess "superweights" massively influencing performance.
- Zeroing one superweight impacts performance more than removing thousands of other important weights.
Emergent Compositionality
- Compositional abilities in AI models emerge multiplicatively, not additively.
- Success at a compound task depends on the product of success rates of individual sub-tasks.
Mixture of Transformers
- Mixture of Transformers (MoT) offers a sparse, scalable architecture for multimodal models.
- MoT routes different modalities to specialized transformers, improving speed and stability.