Exploring Qualitative Evaluations in Language Models and the Path to AGI

This chapter discusses the qualitative evaluation of language models, highlighting the importance of critical reasoning, episodic memory, and collaboration. It emphasizes cautious optimism regarding advancements in language models while acknowledging the challenges that remain on the path to achieving true AGI.

Transcript

chevron_right

Play full episode

chevron_right

Transcript

Episode notes

Episode 133

I spoke with Peter Lee about:

* His early work on compiler generation, metacircularity, and type theory

* Paradoxical problems

* GPT-4s impact, Microsoft’s “Sparks of AGI” paper, and responses and criticism

Enjoy—and let me know what you think!

Peter is President of Microsoft Research. He leads Microsoft Research and incubates new research-powered products and lines of business in areas such as artificial intelligence, computing foundations, health, and life sciences. Before joining Microsoft in 2010, he was at DARPA, where he established a new technology office that created operational capabilities in machine learning, data science, and computational social science. Prior to that, he was a professor and the head of the computer science department at Carnegie Mellon University. Peter is a member of the National Academy of Medicine and serves on the boards of the Allen Institute for Artificial Intelligence, the Brotman Baty Institute for Precision Medicine, and the Kaiser Permanente Bernard J. Tyson School of Medicine. He served on President Obama’s Commission on Enhancing National Cybersecurity. He has testified before both the US House Science and Technology Committee and the US Senate Commerce Committee. With Carey Goldberg and Dr. Isaac Kohane, he is the coauthor of the best-selling book, “The AI Revolution in Medicine: GPT-4 and Beyond.” In 2024, Peter was named by Time magazine as one of the 100 most influential people in health and life sciences.

Find me on Twitter for updates on new episodes, and reach me at editor@thegradient.pub for feedback, ideas, guest suggestions.

I spend a lot of time on this podcast—if you like my work, you can support me on Patreon :) You can also support upkeep for the full Gradient team/project through a paid subscription on Substack!

Subscribe to The Gradient Podcast: Apple Podcasts | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (00:50) Basic vs. applied research

* (05:20) Theory and practice in computing

* (10:28) Traditional denotational semantics and semantics engineering in modern-day systems

* (16:47) Beauty and practicality

* (20:40) Metacircularity in the polymorphic lambda calculus: research directions

* (24:31) Understanding the nature of difficulties with metacircularity

* (26:30) Difficulties with reflection, classic paradoxes

* (31:02) Sparks of AGI

* (31:41) Reproducibility