Machine Learning Street Talk (MLST)

ARC Prize v2 Launch! (Francois Chollet and Mike Knoop)

Mar 24, 2025

Francois Chollet, an AI researcher known for Keras and the ARC challenge, joins Mike Knoop, collaborator on the ARC challenge, to launch the new version of the ARC prize. They discuss how ARC v2 integrates human calibration and adversarial selection, ensuring that even top LLMs struggle against it. The conversation highlights the evolution from ARC v1 to v2, the complexities of AI task design, and the urgent need for rigorous testing methods to bridge the gap between human and AI intelligence in the quest for artificial general intelligence.

54:15

Creator website

Episode guests

Mike Knoop

Francois Chollet

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

The launch of ARC Prize v2 introduces a new benchmark for AI capabilities, calibrated to reflect human difficulty and improve assessment accuracy.

Efficiency in intelligence is emphasized, highlighting the need for AI models to acquire and utilize knowledge effectively rather than solely relying on computational power.

Fluid intelligence is identified as a crucial metric for evaluating AI systems, focusing on adaptability and problem-solving in novel situations.

Deep dives

Introduction of ArcGIS 2 and ArcPrize 2025

The release of ArcGIS 2 marks a significant advancement, providing a new benchmark for assessing the fluid intelligence of AI models. This version serves as an unsaturated benchmark accessible to regular individuals, enabling a clearer measure of AI capabilities. The accompanying ArcPrize 2025 contest encourages innovative contributions, with participants required to open-source their solutions and achieve high efficiency on a Kaggle leaderboard. The structure of this contest builds on last year's success, focusing on novel ideas to push the boundaries of artificial intelligence.

Intro

2min

Reevaluating Intelligence: The Role of Efficiency in AGI

3min

Launch of ARK AGI 2 and ARK Prize 2025 Contest

2min

Bridging the AI Capability Gap

23min

Navigating AI Task Complexity

20min

Exploring the Complexities of Solution Space Prediction in Neural Networks

2min

Understanding Test Time Adaptation in Language Models

2min

Exploring AI Fluid Intelligence and Future Gaps

2min

We are joined by Francois Chollet and Mike Knoop, to launch the new version of the ARC prize! In version 2, the challenges have been calibrated with humans such that at least 2 humans could solve each task in a reasonable task, but also adversarially selected so that frontier reasoning models can't solve them. The best LLMs today get negligible performance on this challenge.

https://arcprize.org/

SPONSOR MESSAGES:

***

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.

Goto https://tufalabs.ai/

***

TRANSCRIPT:

https://www.dropbox.com/scl/fi/0v9o8xcpppdwnkntj59oi/ARCv2.pdf?rlkey=luqb6f141976vra6zdtptv5uj&dl=0

TOC:

1. ARC v2 Core Design & Objectives

[00:00:00] 1.1 ARC v2 Launch and Benchmark Architecture

[00:03:16] 1.2 Test-Time Optimization and AGI Assessment

[00:06:24] 1.3 Human-AI Capability Analysis

[00:13:02] 1.4 OpenAI o3 Initial Performance Results

2. ARC Technical Evolution

[00:17:20] 2.1 ARC-v1 to ARC-v2 Design Improvements

[00:21:12] 2.2 Human Validation Methodology

[00:26:05] 2.3 Task Design and Gaming Prevention

[00:29:11] 2.4 Intelligence Measurement Framework

3. O3 Performance & Future Challenges

[00:38:50] 3.1 O3 Comprehensive Performance Analysis

[00:43:40] 3.2 System Limitations and Failure Modes

[00:49:30] 3.3 Program Synthesis Applications

[00:53:00] 3.4 Future Development Roadmap

REFS:

[00:00:15] On the Measure of Intelligence, François Chollet

https://arxiv.org/abs/1911.01547

[00:06:45] ARC Prize Foundation, François Chollet, Mike Knoop

https://arcprize.org/

[00:12:50] OpenAI o3 model performance on ARC v1, ARC Prize Team

https://arcprize.org/blog/oai-o3-pub-breakthrough

[00:18:30] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, Jason Wei et al.

https://arxiv.org/abs/2201.11903

[00:21:45] ARC-v2 benchmark tasks, Mike Knoop

https://arcprize.org/blog/introducing-arc-agi-public-leaderboard

[00:26:05] ARC Prize 2024: Technical Report, Francois Chollet et al.

https://arxiv.org/html/2412.04604v2

[00:32:45] ARC Prize 2024 Technical Report, Francois Chollet, Mike Knoop, Gregory Kamradt

https://arxiv.org/abs/2412.04604

[00:48:55] The Bitter Lesson, Rich Sutton

http://www.incompleteideas.net/IncIdeas/BitterLesson.html

[00:53:30] Decoding strategies in neural text generation, Sina Zarrieß

https://www.mdpi.com/2078-2489/12/9/355/pdf

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.

Machine Learning Street Talk (MLST)

ARC Prize v2 Launch! (Francois Chollet and Mike Knoop)

Episode guests

Podcast summary created with Snipd AI

Quick takeaways

Deep dives

Introduction of ArcGIS 2 and ArcPrize 2025

Benchmarking Human-Faced Difficulty

Emphasis on Efficiency in Intelligence

The Significance of Fluid Intelligence

The Future of AGI and the Role of Benchmarks

Remember Everything You Learn from Podcasts