Grounded Rewards and Human Motivation in AI

Derby Mill co-host Richard Sutton and his former student, David Silver, recently published a paper about the future of artificial intelligence, called Welcome to the Era of Experience. So in this episode the show’s other hosts—Ajay Agrawal, Sendhil Mullainathan and Niamh Gavin—take their chance to interview Rich about the essay, and provide their take on its implications.

Today’s large language models (LLMs) are trained on human-generated data. So far, this has led to the development of incredible capabilities, such as mastering complex games like backgammon or chess, or absorbing content created by humans and creating fascinating new iterations of art.

While the evolution of LLMs—from AlphaZero (2017) to ChatGPT (2022) to DeepSeek (2025) and beyond—can make it seem as though their possibilities are endless, the agents remain constrained by the scope of the data they are given. In the paper, Silver and Sutton write that “in key domains such as mathematics, coding, and science, the knowledge extracted from human data is rapidly approaching a limit.” Consequently, AI agents will have to be trained on other data, such as their own experiences, which could lead to rapid innovation and superhuman capabilities—a time period which Silver and Sutton refer to as the “age of experience.”

This episode, a roundtable discussion, focuses on the following quotes pulled from the paper:

* Why now? "This will become possible, as outlined above, when agents are able to autonomously act and observe in streams of real-world experience, and where the rewards may be flexibly connected to any of an abundance of grounded, real-world signals."

* Why science? "Perhaps most transformative will be the acceleration of scientific discovery."

* Human-like vs superhuman AIs. "This era of experience will likely be characterised by agents and environments that, in addition to learning from vast quantities of experiential data, will break through the limitations of human-centric AI systems... Furthermore, the pursuit of this agenda by the AI community will spur new innovations in these directions that rapidly progress AI towards truly superhuman agents.”

GUESTS AND HOSTS

Ajay Agrawal, co-founder and partner, Intrepid Growth PartnersRichard Sutton, senior advisor, Intrepid Growth Partners, 2024 Turing Award recipient, pioneer of reinforcement learning and professor, University of AlbertaSendhil Mullainathan, senior advisor, Intrepid Growth Partners, MacArthur Genius grant recipient and professor, MITNiamh Gavin, senior advisor, Intrepid Growth Partners, Applied AI scientist, CEO, Emergent Platforms

LINKS

Derby Mill show websiteRead Rich Sutton’s latest paper Welcome to the Era of ExperienceRich Sutton’s 2019 paper The Bitter LessonCo-founder of OpenAI, Ilya Sutskever, says AI reasoning power will become less predictableListen to our previous episode about DeepSeekCheck out co-author David Silver’s websiteRich Sutton’s home page. Follow Rich on XSendhil Mullainathan’s website. Follow Sendhil on XBe sure to catch every episode of The Derby Mill Series by subscribing on the following platforms: YouTube // Spotify // Apple Podcasts

DISCUSSION POINTS

00:00 Introduction01:34 Context about the paper02:56 Chronology of AI paradigms03:10 Why now?06:09 Niamh’s chronology of AI development14:10 Why science?20:36 Sendhil on scientific research and AI27:07 Grounded vs. ungrounded rewards29:21 Rich on RL temporal difference errors31:10 Human-like vs. superhuman AIs36:40 Final comments

Nugget 01 - AI for Scientific Discovery

Nugget 02 - Is Science like RL?

Nugget 03 - The Value of Experience

DISCLAIMER

The content of this podcast is for informational and educational purposes only and should not be construed as marketing, solicitation, or an offer to buy or sell any securities or investments. The opinions expressed in this video are those of the participants and do not necessarily reflect the views of Intrepid Growth Partners or its affiliates. Any discussion of specific companies, technologies, or industries is for illustrative purposes and does not constitute investment advice. Viewers are encouraged to consult with their own financial, legal, and tax advisors before making any investment decisions.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insights.intrepidgp.com