The Gradient: Perspectives on AI

Peter Lee: Computing Theory and Practice, and GPT-4's Impact

Aug 1, 2024

Peter Lee, President of Microsoft Research, shares insights on the intersection of theory and practice in AI. He recounts his early work on compiler generation and type theory. The conversation dives into the complexities of generative AI, including criticisms of the 'Sparks of AGI' paper and the importance of ethical development. Lee also discusses the significance of robustness in neural networks and the qualitative evaluation of language models, emphasizing the delicate balance between innovation and responsibility in AI advancements.

01:01:48

Creator website

Episode guests

Peter Lee

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Peter Lee emphasizes that the integration of theoretical and practical research in computer science is crucial for advancing the field and addressing contemporary challenges.

He highlights the emotional complexities surrounding AI advancements, stressing the need for transparent communication to mitigate biases in scientific evaluation.

Deep dives

Integration of Theory and Practice in Computer Science

The discussion emphasizes that there is no clear distinction between basic and applied research in computer science, particularly in comparison to fields like physics. Peter Lee reflects on his journey through academia, government, and industry, leading him to conclude that computer science research seamlessly integrates both theoretical and practical elements. He highlights how his views were influenced by observing that theoretical work often requires practical experimentation, and vice versa, underscoring that abstract concepts find their relevance in real-world applications. This notion is further illustrated by the notion of different research modes, including invention and response to contemporary challenges, all of which are crucial for advancing the field.

Intro

5min

The Interplay of Theory and Practice in AI Research

5min

From Skepticism to Semantics

10min

Meta-Theory and Its Paradoxes

10min

Navigating AGI's Complexities

25min

Exploring Qualitative Evaluations in Language Models and the Path to AGI

5min

Episode 133

I spoke with Peter Lee about:

* His early work on compiler generation, metacircularity, and type theory

* Paradoxical problems

* GPT-4s impact, Microsoft’s “Sparks of AGI” paper, and responses and criticism

Enjoy—and let me know what you think!

Peter is President of Microsoft Research. He leads Microsoft Research and incubates new research-powered products and lines of business in areas such as artificial intelligence, computing foundations, health, and life sciences. Before joining Microsoft in 2010, he was at DARPA, where he established a new technology office that created operational capabilities in machine learning, data science, and computational social science. Prior to that, he was a professor and the head of the computer science department at Carnegie Mellon University. Peter is a member of the National Academy of Medicine and serves on the boards of the Allen Institute for Artificial Intelligence, the Brotman Baty Institute for Precision Medicine, and the Kaiser Permanente Bernard J. Tyson School of Medicine. He served on President Obama’s Commission on Enhancing National Cybersecurity. He has testified before both the US House Science and Technology Committee and the US Senate Commerce Committee. With Carey Goldberg and Dr. Isaac Kohane, he is the coauthor of the best-selling book, “The AI Revolution in Medicine: GPT-4 and Beyond.” In 2024, Peter was named by Time magazine as one of the 100 most influential people in health and life sciences.

Find me on Twitter for updates on new episodes, and reach me at editor@thegradient.pub for feedback, ideas, guest suggestions.

I spend a lot of time on this podcast—if you like my work, you can support me on Patreon :) You can also support upkeep for the full Gradient team/project through a paid subscription on Substack!

Subscribe to The Gradient Podcast: Apple Podcasts | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (00:50) Basic vs. applied research

* (05:20) Theory and practice in computing

* (10:28) Traditional denotational semantics and semantics engineering in modern-day systems

* (16:47) Beauty and practicality

* (20:40) Metacircularity in the polymorphic lambda calculus: research directions

* (24:31) Understanding the nature of difficulties with metacircularity

* (26:30) Difficulties with reflection, classic paradoxes

* (31:02) Sparks of AGI

* (31:41) Reproducibility