Stanford Psychology Podcast

122 - Michal Kosinski: Studying Theory of Mind and Reasoning in LLMs.

Nov 30, 2023

Dr. Michal Kosinski, Associate Professor of Organizational Behavior at Stanford University, discusses his research on theory of mind in Large Language Models (LLMs) and reasoning biases. They explore emergent properties in LLMs, the importance of theory of mind in language, testing theory of mind in LLMs, cognitive bias in solving tasks, reasoning vs intuition in language models, and the use of theory of mind tasks in LLMs. They also touch on artificial networks rediscovering human mechanisms and the guest's scientific journey.

01:08:13

Episode guests

Michal Kosinski

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Large language models demonstrate the ability to solve theory of mind tasks, suggesting the possibility of possessing similar capabilities.

Emergent properties like consciousness and understanding can arise in complex systems, exemplified by large language models despite their simplicity compared to the human brain.

Deep dives

Large Language Models and Theory of Mind

Large language models, such as GPT-3.5 Turbo and CHAT GPT-4, demonstrate the ability to solve theory of mind tasks, although caution is needed when interpreting these results. While humans naturally possess theory of mind, the emergence of this ability in artificial neural networks suggests the possibility that language-trained neural networks could also possess similar capabilities. However, due to the complexity and unique nature of human consciousness and understanding, further research is needed to fully understand the presence and extent of theory of mind in large language models.

Introduction

2min

Emergent Properties in Large Language Models

4min

The Importance of Theory of Mind in Human Language

11min

Testing Theory of Mind in Large Language Models

17min

Cognitive Bias in Solving a Pool Filling Task

2min

Reasoning vs Intuition in Language Models

17min

Theory of Mind Tasks in Large Language Models

3min

Artificial Networks and Human Psychology

12min

Xi Jia chats with Dr. Michal Kosinski, an Associate Professor of Organizational Behavior at Stanford University's Graduate School of Business. Michal's research interests recently encompass both human and artificial cognition. Currently, his work centers on examining the psychological processes in Large Language Models (LLMs), and leveraging Artificial Intelligence (AI), Machine Learning (ML), Big Data, and computational techniques to model and predict human behavior.

In this episode, they chat about Michal's recent works: "Theory of Mind Might Have Spontaneously Emerged in Large Language Models" and "Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in ChatGPT". Michal also shared his scientific journey and some personal suggestions for PhD students.

If you found this episode interesting at all, subscribe on our Substack and consider leaving us a good rating! It just takes a second but will allow us to reach more people and make them excited about psychology.

Michal's paper on Theory of Mind in LLMs: https://arxiv.org/abs/2302.02083
Michal's paper on reasoning bias in LLMs: https://www.nature.com/articles/s43588-023-00527-x

Michal's personal website: https://www.michalkosinski.com/

Xi Jia's profile: https://profiles.stanford.edu/xijia-zhou

Xi Jia's Twitter/X: https://twitter.com/LauraXijiaZhou

Podcast Twitter @StanfordPsyPod

Podcast Substack https://stanfordpsypod.substack.com/

Let us know what you thought of this episode, or of the podcast! :) stanfordpsychpodcast@gmail.com