Unsupervised Learning Ep 58: Google Researchers Noam Shazeer and Jack Rae on Scaling Test-time Compute, Reactions to Ilya & AGI
134 snips
Mar 17, 2025 
Noam Shazeer, co-inventor of the Transformer, and Jack Rae, Research Director at DeepMind, dive into the future of AI. They discuss the groundbreaking capabilities of Gemini 2.0 for reasoning and creativity. The duo explores the complexities of AI evaluation metrics and the evolving role of test-time compute, emphasizing efficiency over traditional methods. They also reflect on the philosophical challenges of AGI, the rise of vision-based models, and AI's transformative impact on healthcare and education, highlighting the balance between innovation and safety.
AI Snips
Chapters
Transcript
Episode notes
Test-Time Compute Generality
- Test-time compute was initially focused on reasoning tasks like math and code.
- Surprisingly, it also improved creative tasks like essay writing, showing an unexpected generality.
Benchmark Importance
- Good benchmarks are crucial for driving progress in AI research, even if focusing on specific areas like math.
- They help distinguish genuine reasoning ability from simply memorizing data and increasing perplexity.
Evaluation Saturation
- Meaningful evaluations are hard to find because models quickly saturate them.
- What's considered challenging today might become trivial in a few months, making previously important benchmarks obsolete.