Is There an Analog of the Distributional Hypothesis?

3min Snip

00:00

Play full episode

Summary

Transcript

Episode notes

Wittgenstein said that the semantics of language are in large part indistinguishable from their pragmatics by how we use them. To truly understand language you need to incorporate some aspect of pragmatics, a pragmatic understanding. And so this motivated me to do two things across the last 10 years. First is to move away from the Dr. Ork I focused on to work on a class of methods that were more scalable but also gave greater freedom to incorporate information from non-linguistic channels or from other parts of the environment than just the words. In turn, having to study agents within such contexts would lead us to have a better understanding of the sort of semantics that emerged within the

In episode 62 of The Gradient Podcast, Daniel Bashir speaks to Ed Grefenstette.

Ed is Head of Machine Learning at Cohere and an Honorary Professor at University College London. He previously held research scientist positions at Facebook AI Research and DeepMind, following a stint as co-founder and CTO of Dark Blue Labs. Before his time in industry, Ed worked at Oxford’s Department of Computer Science as a lecturer and Fulford Junior Research Fellow at Somerville College. Ed also received his MSc and DPhil from Oxford’s Computer Science Department.

Have suggestions for future podcast guests (or other feedback)? Let us know here!

Subscribe to The Gradient Podcast: Apple Podcasts | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:18) The Ed Grefenstette Origin Story

* (08:15) Distributional semantics and Ed’s PhD research

* (14:30) Extending the distributional hypothesis, later Wittgenstein

* (18:00) Recovering parse trees in LMs, can LLMs understand communication and not just bare language?

* (23:15) LMs capture something about pragmatics, proxies for grounding and pragmatics

* (25:00) Human-in-the-loop training and RLHF—what is the essential differentiator?

* (28:15) A convolutional neural network for modeling sentences, relationship to attention

* (34:20) Difficulty of constructing supervised learning datasets, benchmark-driven development