Machine Learning Street Talk (MLST)

#84 LAURA RUIS - Large language models are not zero-shot communicators [NEURIPS UNPLUGGED]

Dec 6, 2022

In this insightful discussion, Laura Ruis, a researcher focused on pragmatic inferences in conversational AI, delves into the limitations of large language models. She reveals how these models struggle with context and implicature, causing misunderstandings in communication. Ruis also examines zero-shot learning capabilities, showcasing disparities in performance across different models. Additionally, she highlights the importance of human feedback in refining these AI systems, aiming for a future where they can more effectively interpret and engage in nuanced conversations.

27:47

Creator website

Episode guests

Laura Ruis

AI Chapters

Episode notes

Intro

4min

Exploring the Communication Limitations of Large Language Models

2min

Exploring Zero-Shot Learning in Language Models

3min

Reinforcement Learning and the Alignment of Language Models

3min

Understanding Language Models: Nuances and Limitations

14min

Exploring Fairness and Creativity in Language Models

2min

In this NeurIPSs interview, we speak with Laura Ruis about her research on the ability of language models to interpret language in context. She has designed a simple task to evaluate the performance of widely used state-of-the-art language models and has found that they struggle to make pragmatic inferences (implicatures). Tune in to learn more about her findings and what they mean for the future of conversational AI.

Laura Ruis

https://www.lauraruis.com/

https://twitter.com/LauraRuis

BLOOM

https://bigscience.huggingface.co/blog/bloom

Large language models are not zero-shot communicators [Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, Edward Grefenstette]

https://arxiv.org/abs/2210.14986

[Zhang et al] OPT: Open Pre-trained Transformer Language Models

https://arxiv.org/pdf/2205.01068.pdf

[Lampinen] Can language models handle recursively nested grammatical structures? A case study on comparing models and humans

https://arxiv.org/pdf/2210.15303.pdf

[Gary Marcus] Horse rides astronaut

https://garymarcus.substack.com/p/horse-rides-astronaut

[Gary Marcus] GPT-3, Bloviator: OpenAI’s language generator has no idea what it’s talking about

https://www.technologyreview.com/2020/08/22/1007539/gpt3-openai-language-generator-artificial-intelligence-ai-opinion/

[Bender et al] On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?

https://dl.acm.org/doi/10.1145/3442188.3445922

[janus] Simulators (Less Wrong)

https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.