Efficiency in Acquiring New Skills and ARC Benchmark

This chapter explores the concept of artificial general intelligence (AGI) and discusses the ARC benchmark designed to measure efficiency in acquiring new skills. It delves into the importance of testing systems with novel tasks to assess true intelligence and emphasizes the significance of open progress and science in AGI research.

Transcript

chevron_right

Play full episode

chevron_right

Transcript

Episode notes

As impressive as LLMs are, the growing consensus is that language, scale and compute won’t get us to AGI. Although many AI benchmarks have quickly achieved human-level performance, there is one eval that has barely budged since it was created in 2019.

Google researcher François Chollet wrote a paper that year defining intelligence as skill-acquisition efficiency—the ability to learn new skills as humans do, from a small number of examples. To make it testable he proposed a new benchmark, the Abstraction and Reasoning Corpus (ARC), designed to be easy for humans, but hard for AI. Notably, it doesn’t rely on language.

Zapier co-founder Mike Knoop read Chollet’s paper as the LLM wave was rising. He worked quickly to integrate generative AI into Zapier’s product, but kept coming back to the lack of progress on the ARC benchmark. In June, Knoop and Chollet launched the ARC Prize, a public competition offering more than $1M to beat and open-source a solution to the ARC-AGI eval.

In this episode Mike talks about the new ideas required to solve ARC, shares updates from the first two weeks of the competition, and shares why he’s excited for AGI systems that can innovate alongside humans.

Hosted by: Sonya Huang and Pat Grady, Sequoia Capital

Mentioned:

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models: The 2019 paper that first caught Mike’s attention about the capabilities of LLMs
On the Measure of Intelligence: 2019 paper by Google researcher François Chollet that introduced the ARC benchmark, which remains unbeaten
ARC Prize 2024: The $1M+ competition Mike and François have launched to drive interest in solving the ARC-AGI eval
Sequence to Sequence Learning with Neural Networks: Ilya Sutskever paper from 2014 that influenced the direction of machine translation with deep neural networks.
Etched: Luke Miles on LessWrong wrote about the first ASIC chip that accelerates transformers on silicon
Kaggle: The leading data science competition platform and online community, acquired by Google in 2017
Lab42: Swiss AU lab that hosted ARCathon precursor to ARC Prize
Jack Cole: Researcher on team that was #1 on the leaderboard for ARCathon
Ryan Greenblatt: Researcher with current high score (50%) on ARC public leaderboard