The MAD Podcast with Matt Turck

Chasing Real AGI: Inside ARC Prize 2025 with Chollet & Knoop

16 snips
Apr 3, 2025
Join François Chollet, a leading AI researcher and philosopher behind Keras, and Mike Knoop, co-founder of Zapier and the ARC Prize, as they explore the future of artificial general intelligence (AGI). They discuss the launch of ARC AGI 2 and how current language models fall short in true intelligence benchmarks. Tune in to hear about advancements like the O3 model, the significance of test-time adaptation, and the philosophical underpinnings of intelligence. Their new research lab, Ndea, aims to revolutionize AI and foster rapid scientific progress.
Ask episode
AI Snips
Chapters
Transcript
Episode notes
INSIGHT

ARC's Focus: Fluid Intelligence

  • ARC measures AI fluid intelligence, unlike other benchmarks that focus on specific skills.
  • It uses novel tasks to evaluate how well AI adapts, a current weakness.
INSIGHT

Shift to Test-Time Adaptation

  • The AI research world shifted from static LLMs to test-time adaptation.
  • Models like O3, with longer latency and higher cost, show stronger generalization.
ANECDOTE

LLM Scaling Limitations

  • From GPT-2 to GPT-4, a 50,000x scale-up resulted in almost no improvement on ARC.
  • ARC resists the pre-training scaling paradigm, highlighting the need for different approaches.
Get the Snipd Podcast app to discover more snips from this episode
Get the app