Is There a Scientific Approach to Machine Learning?

5min Snip

00:00

Play full episode

Summary

Transcript

Episode notes

Yasha: These questions of how do these models work and really taking a very scientific approach to it is incredibly important and valuable. Yasha was one of the pioneers in deep learning for language modeling or the machine translation already in 1998. And then he realized that in particular for the causal discovery, it's really important to consider all possible ways in which the causal mechanism could have given rise to the data observed. There why we need to marginalize them out now instead of just focusing on GPT scale models.

In episode 59 of The Gradient Podcast, Daniel Bashir speaks to Professor Kyunghyun Cho.

Professor Cho is an associate professor of computer science and data science at New York University and CIFAR Fellow of Learning in Machines & Brains. He is also a senior director of frontier research at the Prescient Design team within Genentech Research & Early Development. He was a research scientist at Facebook AI Research from 2017-2020 and a postdoctoral fellow at University of Montreal under the supervision of Prof. Yoshua Bengio after receiving his MSc and PhD degrees from Aalto University. He received the Samsung Ho-Am Prize in Engineering in 2021.

Have suggestions for future podcast guests (or other feedback)? Let us know here!

Subscribe to The Gradient Podcast: Apple Podcasts | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:15) How Professor Cho got into AI, going to Finland for a PhD

* (06:30) Accidental and non-accidental parts of Prof Cho’s journey, the role of timing in career trajectories

* (09:30) Prof Cho’s M.Sc. thesis on Restricted Boltzmann Machines

* (17:00) The state of autodiff at the time

* (20:00) Finding non-mainstream problems and examining limitations of mainstream approaches, anti-dogmatism, Yoshua Bengio appreciation

* (24:30) Detaching identity from work, scientific training

* (26:30) The rest of Prof Cho’s PhD, the first ICLR conference, working in Yoshua Bengio’s lab

* (34:00) Prof Cho’s isolation during his PhD and its impact on his work—transcending insecurity and working on unsexy problems

* (41:30) The importance of identifying important problems and developing an independent research program, ceiling on the number of important research problems

* (46:00) Working on Neural Machine Translation, Jointly Learning to Align and Translate

* (1:01:45) What RNNs and earlier NN architectures can still teach us, why transformers were successful

* (1:08:00) Science progresses gradually

* (1:09:00) Learning distributed representations of sentences, extending the distributional hypothesis

* (1:21:00) Difficulty and limitations in evaluation—directions of dynamic benchmarks, trainable evaluation metrics

* (1:29:30) Mixout and AdapterFusion: fine-tuning and intervening on pre-trained models, pre-training as initialization, destructive interference

* (1:39:00) Analyzing neural networks as reading tea leaves

* (1:44:45) Importance of healthy skepticism for scientists

* (1:45:30) Language-guided policies and grounding, vision-language navigation

* (1:55:30) Prof Cho’s reflections on 2022

* (2:00:00) Obligatory ChatGPT content

* (2:04:50) Finding balance