What Are the Major Ingredients of Transformers?

6min Snip

00:00

Play full episode

Summary

Transcript

Episode notes

RNNs and LSTMs are taught as kind of like this is maybe a historical artifact in deep learning courses. Do you think there are insights remaining to be gleamed from RNNs and these earlier architectures, despite the fact that NLP now seems to be all about transformers? Oh yeah, absolutely. And it seems you were right. I do have a few questions about this set of papers. The first is just thinking through the evolution of things over time. After attention got taken forward and really taken to its logical endpoint with the development of the transformer. But as entities in themselves, at least from my perspective, I don't see nearly as much attention being paid to them

In episode 59 of The Gradient Podcast, Daniel Bashir speaks to Professor Kyunghyun Cho.

Professor Cho is an associate professor of computer science and data science at New York University and CIFAR Fellow of Learning in Machines & Brains. He is also a senior director of frontier research at the Prescient Design team within Genentech Research & Early Development. He was a research scientist at Facebook AI Research from 2017-2020 and a postdoctoral fellow at University of Montreal under the supervision of Prof. Yoshua Bengio after receiving his MSc and PhD degrees from Aalto University. He received the Samsung Ho-Am Prize in Engineering in 2021.

Have suggestions for future podcast guests (or other feedback)? Let us know here!

Subscribe to The Gradient Podcast: Apple Podcasts | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:15) How Professor Cho got into AI, going to Finland for a PhD

* (06:30) Accidental and non-accidental parts of Prof Cho’s journey, the role of timing in career trajectories

* (09:30) Prof Cho’s M.Sc. thesis on Restricted Boltzmann Machines

* (17:00) The state of autodiff at the time

* (20:00) Finding non-mainstream problems and examining limitations of mainstream approaches, anti-dogmatism, Yoshua Bengio appreciation

* (24:30) Detaching identity from work, scientific training

* (26:30) The rest of Prof Cho’s PhD, the first ICLR conference, working in Yoshua Bengio’s lab

* (34:00) Prof Cho’s isolation during his PhD and its impact on his work—transcending insecurity and working on unsexy problems

* (41:30) The importance of identifying important problems and developing an independent research program, ceiling on the number of important research problems

* (46:00) Working on Neural Machine Translation, Jointly Learning to Align and Translate

* (1:01:45) What RNNs and earlier NN architectures can still teach us, why transformers were successful

* (1:08:00) Science progresses gradually

* (1:09:00) Learning distributed representations of sentences, extending the distributional hypothesis

* (1:21:00) Difficulty and limitations in evaluation—directions of dynamic benchmarks, trainable evaluation metrics

* (1:29:30) Mixout and AdapterFusion: fine-tuning and intervening on pre-trained models, pre-training as initialization, destructive interference

* (1:39:00) Analyzing neural networks as reading tea leaves

* (1:44:45) Importance of healthy skepticism for scientists

* (1:45:30) Language-guided policies and grounding, vision-language navigation

* (1:55:30) Prof Cho’s reflections on 2022

* (2:00:00) Obligatory ChatGPT content

* (2:04:50) Finding balance