The Gradient: Perspectives on AI

Sasha Rush: Building Better NLP Systems

Feb 29, 2024

Professor Sasha Rush discusses the importance of learning and inference in AI, state-space models as an alternative to Transformers, efficiency enhancements in NLP systems through techniques like sequence level knowledge distillation, and the evolution of research perspective towards empirical approaches in NLP systems.

54:03

Creator website

Episode guests

Sasha Rush

AI Summary

Highlights

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Sasha Rush's research focuses on developing safe and controllable NLP systems with efficient algorithms for text generation.

He explores state-space models as a potential alternative to Transformers in NLP architectures, emphasizing innovative approaches beyond existing models.

Deep dives

Sasha Rush's Journey in NLP and Language Tasks

Sasha Rush's extensive experience in text generation and efficient algorithms for language tasks is highlighted. His transition from syntax-focused research to exploration of NLP through optimization was discussed, showcasing his intricate journey and evolution in the field.

Exploring Controllability and Attribute Ability in NLP Systems

02:45

Significance of Low-Level Interfaces in PyTorch Development

01:34

Exploring Engineering in AI, Funding Distribution, and a New Language Modeling Conference

03:39

Efficient State Space Models as Alternatives to Transformers

01:40

Introduction

2min

Evolution of NLP: From Optimization to Learning

9min

Fostering a Diverse and Supportive Research Environment

2min

Exploring State Space Models in NLP Research

8min

Efficiency Enhancement in Natural Language Processing Systems

8min

Exploring Prom Tuning and Controllability in NLP Systems

13min

Evolving Deep Learning Tools and Interfaces

9min

Evolution of Research Perspective Towards Empirical Approaches in NLP Systems

4min

In episode 113 of The Gradient Podcast, Daniel Bashir speaks to Professor Sasha Rush.

Professor Rush is an Associate Professor at Cornell University and a Researcher at HuggingFace. His research aims to develop natural language processing systems that are safe, fast, and controllable. His group is interested primarily in tasks that involve text generation, and they study data-driven probabilistic methods that combine deep-learning based models with probabilistic controls. He is also interested in open-source NLP and deep learning, and develops projects to make deep learning systems safer, clearer, and easier to use.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at editor@thegradient.pub

Subscribe to The Gradient Podcast: Apple Podcasts | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (01:47) Professor Rush’s background

* (03:23) Professor Rush’s reflections on prior work—importance of learning and inference

* (04:58) How much engineering matters in deep learning, the Rush vs. Frankle Bet

* (07:12) On encouraging and incubating good research

* (10:50) Features of good research environments

* (12:36) 5% bets in Professor Rush’s research: State-Space Models (SSMs) as an alternative to Transformers

* (15:58) SSMs vs. Transformers

* (18:53) Probabilistic Context-Free Grammars—are (P)CFGs worth paying attention to?