Skepticism and the Path to AGI: Evaluating LLM Performance

Here is my conversation with Francois Chollet and Mike Knoop on the $1 million ARC-AGI Prize they're launching today.

I did a bunch of socratic grilling throughout, but Francois’s arguments about why LLMs won’t lead to AGI are very interesting and worth thinking through.

It was really fun discussing/debating the cruxes. Enjoy!

Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here.

Timestamps

(00:00:00) – The ARC benchmark

(00:11:10) – Why LLMs struggle with ARC

(00:19:00) – Skill vs intelligence

(00:27:55) - Do we need “AGI” to automate most jobs?

(00:48:28) – Future of AI progress: deep learning + program synthesis

(01:00:40) – How Mike Knoop got nerd-sniped by ARC

(01:08:37) – Million $ ARC Prize

(01:10:33) – Resisting benchmark saturation

(01:18:08) – ARC scores on frontier vs open source models

(01:26:19) – Possible solutions to ARC Prize