The MAD Podcast with Matt Turck cover image

The MAD Podcast with Matt Turck

Beyond Brute Force: Chollet & Knoop on ARC AGI 2, the Benchmark Breaking LLMs and the Search for True Machine Intelligence

Apr 3, 2025
Join François Chollet, a leading AI researcher and philosopher behind Keras, and Mike Knoop, co-founder of Zapier and the ARC Prize, as they explore the future of artificial general intelligence (AGI). They discuss the launch of ARC AGI 2 and how current language models fall short in true intelligence benchmarks. Tune in to hear about advancements like the O3 model, the significance of test-time adaptation, and the philosophical underpinnings of intelligence. Their new research lab, Ndea, aims to revolutionize AI and foster rapid scientific progress.
01:00:45

Podcast summary created with Snipd AI

Quick takeaways

  • Current LLMs excel at specialized tasks but struggle with real-time adaptation, highlighting the evolving definition of AGI.
  • The new ARC AGI 2 benchmark focuses on fluid intelligence and emphasizes genuine problem-solving over brute-force solutions.

Deep dives

Limitations and Evolution of LLMs

Current large language models (LLMs) excel at processing vast amounts of information and performing specialized tasks, but they struggle with adapting to new and novel situations in real-time. This limitation underscores the evolving definition of artificial general intelligence (AGI), which is characterized by an AI's ability to tackle tasks that humans can perform naturally, yet the AI cannot. The discussion highlights a shift in AI research, moving away from merely scaling up models like GPT-3 and GPT-4, which operate statically during inference. Instead, the focus is turning toward test-time adaptation and program synthesis as promising pathways toward achieving AGI.

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.
App store bannerPlay store banner

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode