Advancing AI Systems for Increased Reliability and Innovation

As impressive as LLMs are, the growing consensus is that language, scale and compute won’t get us to AGI. Although many AI benchmarks have quickly achieved human-level performance, there is one eval that has barely budged since it was created in 2019.

Google researcher François Chollet wrote a paper that year defining intelligence as skill-acquisition efficiency—the ability to learn new skills as humans do, from a small number of examples. To make it testable he proposed a new benchmark, the Abstraction and Reasoning Corpus (ARC), designed to be easy for humans, but hard for AI. Notably, it doesn’t rely on language.

Zapier co-founder Mike Knoop read Chollet’s paper as the LLM wave was rising. He worked quickly to integrate generative AI into Zapier’s product, but kept coming back to the lack of progress on the ARC benchmark. In June, Knoop and Chollet launched the ARC Prize, a public competition offering more than $1M to beat and open-source a solution to the ARC-AGI eval.

In this episode Mike talks about the new ideas required to solve ARC, shares updates from the first two weeks of the competition, and shares why he’s excited for AGI systems that can innovate alongside humans.

Hosted by: Sonya Huang and Pat Grady, Sequoia Capital 

Mentioned:

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models: The 2019 paper that first caught Mike’s attention about the capabilities of LLMs
• On the Measure of Intelligence: 2019 paper by Google researcher François Chollet that introduced the ARC benchmark, which remains unbeaten
• ARC Prize 2024: The $1M+ competition Mike and François have launched to drive interest in solving the ARC-AGI eval
• Sequence to Sequence Learning with Neural Networks: Ilya Sutskever paper from 2014 that influenced the direction of machine translation with deep neural networks.
• Etched: Luke Miles on LessWrong wrote about the first ASIC chip that accelerates transformers on silicon
• Kaggle: The leading data science competition platform and online community, acquired by Google in 2017
• Lab42: Swiss AU lab that hosted ARCathon precursor to ARC Prize
• Jack Cole: Researcher on team that was #1 on the leaderboard for ARCathon
• Ryan Greenblatt: Researcher with current high score (50%) on ARC public leaderboard

(00:00) Introduction

(01:51) AI at Zapier

(08:31) What is ARC AGI?

(13:25) What does it mean to efficiently acquire a new skill?

(19:03) What approaches will succeed?

(21:11) A little bit of a different shape

(25:59) The role of code generation and program synthesis

(29:11) What types of people are working on this?

(31:45) Trying to prove you wrong

(34:50) Where are the big labs?

(38:21) The world post-AGI

(42:51) When will we cross 85% on ARC AGI?

(46:12) Will LLMs be part of the solution?

(50:13) Lightning round