Interconnects

Interviewing Finbarr Timbers on the "We are So Back" Era of Reinforcement Learning

Dec 5, 2024

Finbarr Timbers, an AI researcher with a background at DeepMind and Midjourney, dives deep into the world of reinforcement learning. He explains the evolution of RL, from fundamental algorithms to its resurgence with breakthroughs like AlphaZero and ChatGPT. Timbers shares stories about teaching AI to tackle Atari games and discusses modern advancements in natural language processing. He highlights the growing importance of data annotation in RL and contrasts the pressure of deadlines in tech with lessons from endurance sports, emphasizing innovation.

01:08:33

Creator website

Episode guests

Finbarr Timbers

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Reinforcement learning (RL) is deeply rooted in Markov Decision Processes, emphasizing the balance between exploration and exploitation in complex tasks.

The evolution of deep reinforcement learning showcases pivotal advancements, from DQN for Atari to significant cultural milestones like AlphaGo.

The integration of RL from human feedback (RLHF) with language models is reshaping their development and expanding the scope of RL applications.

Deep dives

Understanding Reinforcement Learning

Reinforcement learning (RL) is defined through the classic framework of Markov Decision Processes (MDPs), where agents navigate through states, take actions, and receive rewards. The key to RL lies in the exploration versus exploitation trade-off, especially in tasks requiring significant exploration like learning to play Atari games from scratch. This contrasts with supervised learning, where prior knowledge guides the process. Additionally, the discussion reveals how RL often overlaps with recent advancements in language models, noting that some popular applications may not fully utilize the core principles of RL.

Intro

3min

Navigating Reinforcement Learning: From Economics to AI

8min

Evolving Dynamics of Reinforcement Learning and NLP

22min

Navigating Reinforcement Learning in Search Algorithms

11min

The Rising Importance of Data Annotation in Reinforcement Learning

2min

Navigating AI Development Challenges

21min

The Pressure of Deadlines: Lessons from Endurance Sports and Corporate Innovation

2min

Finbarr Timbers is an AI researcher who writes Artificial Fintelligence — one of the technical AI blog’s I’ve been recommending for a long time — and has a variety of experiences at top AI labs including DeepMind and Midjourney. The goal of this interview was to do a few things:

* Revisit what reinforcement learning (RL) actually is, its origins, and its motivations.

* Contextualize the major breakthroughs of deep RL in the last decade, from DQN for Atari to AlphaZero to ChatGPT. How could we have seen the resurgence coming? (see the timeline below for the major events we cover)

* Modern uses for RL, o1, RLHF, and the future of finetuning all ML models.

* Address some of the critiques like “RL doesn’t work yet.”

It was a fun one. Listen on Apple Podcasts, Spotify, YouTube, and where ever you get your podcasts. For other Interconnects interviews, go here.

Timeline of RL and what was happening at the time

In the last decade of deep RL, there have been a few phases.

* Era 1: Deep RL fundamentals — when modern algorithms we designed and proven.

* Era 2: Major projects — AlphaZero, OpenAI 5, and all the projects that put RL on the map.

* Era 3: Slowdown — when DeepMind and OpenAI no longer had the major RL projects and cultural relevance declined.

* Era 4: RLHF & widening success — RL’s new life post ChatGPT.

Covering these is the following events. This is incomplete, but enough to inspire a conversation.

Early era: TD Gammon, REINFORCE, Etc

2013: Deep Q Learning (Atari)