The Derby Mill Series Are LLMs Bitter Lesson Pilled? (The Derby Mill Series ep 18)
A trillion-dollar clash of ideas is roiling the artificial intelligence community. Today, in a special episode, our host Ajay Agrawal leads Rich Sutton, Sendhil Mullainathan and Niamh Gavin, and special guest Suzanne Gildert, in a fascinating exploration of the issue: Are Large Language Models (LLMs) sufficiently “bitter lesson pilled” to live up to their hype?
“Bitter lesson pilled” is the AI community’s term of art for scaling with the constantly falling cost of compute (e.g., search and learning). The term arises from Rich Sutton’s 2019 essay, The Bitter Lesson.
As he recently told independent journalist Dwarkesh Patel on the Dwarkesh Podcast, Rich Sutton does not believe that LLMs are sufficiently “bitter lesson pilled.” In other words, Rich believes LLMs suffer from a key vulnerability: A limit exists on their ability to improve – and it’s much closer than we’ve been led to believe.
GUESTS AND HOSTS
Ajay Agrawal, co-founder and partner, Intrepid Growth PartnersRichard Sutton, senior advisor, Intrepid Growth Partners, 2024 Turing Award recipient, pioneer of reinforcement learning and professor, University of AlbertaSendhil Mullainathan, senior advisor, Intrepid Growth Partners, MacArthur Genius grant recipient and professor, MITNiamh Gavin, senior advisor, Intrepid Growth Partners, Applied AI scientist, CEO, Emergent PlatformsSuzanne Gildert, founder and CEO, Nirvanic Consciousness Technologies
LINKS
The Dwarkesh Podcast episode featuring Rich Sutton. The computer scientist Andrej Karpathy’s take. Rich’s original Bitter Lesson essay.Meta machine-learning engineer Chris Hayduk’s tweet about the debate on X, retweeted by Rich and referenced in this episode by Sendhil.Good description of the train-fly problem that Sendhil mentioned, from Presh Talwalkar. Derby Mill series website. Derby Mill is created by the team at Intrepid Growth Partners.Rich Sutton’s home page. Follow Rich on XSendhil Mullainathan’s website. Follow Sendhil on XBe sure to catch every episode of The Derby Mill Series by subscribing on the following platforms:YouTube // Spotify // Apple Podcasts
DISCUSSION POINTS
00:00 Cold open00:39 Context for episode01:39 The bitter Lesson02:49 Supervised learning04:30 Challenge of RL09:49 Discussing a Tweet13:30 Rich’s opinion on the big lesson21:28 Tension in the LLM space23:25 Behaviour and extrapolation25:27 What is considered AI26:05 Final remarks
NUGGETS
Why Squirrels Still Outthink Supervised AI (1801)Derby Mill Series host Ajay Agrawal asks co-host Suzanne Gildert, why can’t AI learn like a squirrel?
Addressing Rich’s Tweet (1802)MacArthur Genius Award recipient Sendhil Mullainathan responds to a tweet that underscores a key difference between LLMs and humans.
What Happens if LLMs Don’t Pay Off Soon (1803)The Bitter Lesson says, “look out if you’re putting all your eggs into the basket of human knowledge,” according to Turing Award recipient Richard Sutton.
DISCLAIMER
The content of this podcast is for informational and educational purposes only and should not be construed as marketing, solicitation, or an offer to buy or sell any securities or investments. The opinions expressed in this video are those of the participants and do not necessarily reflect the views of Intrepid Growth Partners or its affiliates. Any discussion of specific companies, technologies, or industries is for illustrative purposes and does not constitute investment advice. Viewers are encouraged to consult with their own financial, legal, and tax advisors before making any investment decisions.
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insights.intrepidgp.com