AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Uncovering Indirect Relationships through Text Analysis
Language models like GPT-3 can infer indirect relationships from text but struggle to explain the full human effect. With sufficient text data and computing power, these models can achieve sensitivity levels previously thought unlikely. However, the exploration of how other modalities like images can complement text analysis is also crucial for further advancements in understanding complex relationships.
In episode 112 of The Gradient Podcast, Daniel Bashir speaks to Cameron Jones and Sean Trott.
Cameron is a PhD candidate in the Cognitive Science Department at the University of California, San Diego. His research compares how humans and large language models process language about world knowledge, situation models, and theory of mind.
Sean is an Assistant Teaching Professor in the Cognitive Science Department at the University of California, San Diego. His research interests include probing large language models, ambiguity in languages, how ambiguous words are represented, and pragmatic inference. He previously completed his PhD at UCSD.
Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at editor@thegradient.pub
Subscribe to The Gradient Podcast: Apple Podcasts | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter
Outline:
* (00:00) Intro
* (02:55) Cameron’s background
* (06:00) Sean’s background
* (08:15) Unexpected capabilities of language models and the need for embodiment to understand meaning
* (11:05) Interpreting results of Turing tests, separating what humans and LLMs do when behaving as though they “understand”
* (14:27) Internal mechanisms, interpretability, how we test theories
* (16:40) Languages are efficient, but for whom?
* (17:30) Initial motivations: lexical ambiguity
* (19:20) The balance of meanings across wordforms
* (22:35) Tension between speaker- and comprehender-oriented pressures in lexical ambiguity
* (25:05) Context and potential vs. realized ambiguity
* (27:15) LLM-ology
* (28:30) Studying LLMs as models of human cognition and as interesting objects of study in their own right
* (30:03) Example of explaining away effects
* (33:54) The internalist account of belief sensitivity—behavior and internal representations
* (37:43) LLMs and the False Belief Task
* (42:05) Hypothetical on observed behavior and inferences about internal representations
* (48:05) Distributional Semantics Still Can’t Account for Affordances
* (50:25) Tests of embodied theories and limitations of distributional cues
* (53:54) Multimodal models and object affordances
* (58:30) Language and grounding, other buzzwords
* (59:45) How could we know if LLMs understand language?
* (1:04:50) Reference: as a thing words do vs. ontological notion
* (1:11:38) The Role of Physical Inference in Pronoun Resolution
* (1:16:40) World models and world knowledge
* (1:19:45) EPITOME
* (1:20:20) The different tasks
* (1:26:43) Confounders / “attending” in LM performance on tasks
* (1:30:30) Another hypothetical, on theory of mind
* (1:32:26) How much information can language provide in service of mentalizing?
* (1:35:14) Convergent validity and coherence/validity of theory of mind
* (1:39:30) Interpretive questions about behavior w/r/t/ theory of mind
* (1:43:35) Does GPT-4 Pass the Turing Test?
* (1:44:00) History of the Turing Test
* (1:47:05) Interrogator strategies and the strength of the Turing Test
* (1:52:15) “Internal life” and personality
* (1:53:30) How should this research impact how we assess / think about LLM abilities?
* (1:58:56) Outro
Links:
* Cameron’s homepage and Twitter
* Research — Language and NLP
* Languages are efficient, but for whom?
* Research — LLM-ology
* Do LLMs know what humans know?
* Distributional Semantics Still Can’t Account for Affordances
* In Cautious Defense of LLM-ology
* Should Psycholinguists use LLMs as “model organisms”?
* (Re)construing Meaning in NLP
* Research — language and grounding, theory of mind, reference [insert other buzzwords here]
* Do LLMs have a “theory of mind”?
* How could we know if LLMs understand language?
* Does GPT-4 Pass the Turing Test?
* The extended mind and why it matters for cognitive science research
* EPITOME
* The Role of Physical Inference in Pronoun Resolution
Listen to all your favourite podcasts with AI-powered features
Listen to the best highlights from the podcasts you love and dive into the full episode
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
Listen to all your favourite podcasts with AI-powered features
Listen to the best highlights from the podcasts you love and dive into the full episode