Generally Intelligent

Episode 20: Hattie Zhou, Mila, on supermasks, iterative learning, and fortuitous forgetting

Oct 14, 2022

01:47:28

Creator website

Episode guests

Hattie Zhou

Highlights

AI Chapters

Episode notes

The Lottery Ticket Hypothesis for Deep Learning

02:00

What Is a Not-Correlational?

02:32

Retraining Models to Steer Towards Desirable Characteristics

02:58

Getting Rid of Specifics for Spurious Features

01:37

Introduction

2min

Is the Model Doing Some Kind of Implicit Curriculum Learning While It's Being Trained?

2min

Is There a Trick for Generalization?

3min

How to Get an Extremely Adversarial Initialization

2min

The Lottery Ticket Hypothesis

2min

How to Train a Sparse Network?

2min

The Super Mask

5min

Is There a Way to Change Architectures?

4min

How to Find Sub-Networks That Are the Right Shape of the Solution?

4min

10.

Is There a Sparse Architecture in Computer Vision?

2min

11.

What's the Biggest Takeaway From the Lottery Ticket Hypothesis Paper?

2min

12.

Is There a Better Way to Train Super Masks?

2min

13.

Is There a Way to Control the Behavior of Pre-Trained Models?

3min

14.

What Is a Not-Correlational?

3min

15.

The Zero Values Are Relevant Still. Is That Really a Good Idea?

5min

16.

The Story of Coherent Gradients

2min

17.

Tendexure, I Love That!

3min

18.

Increasing Compositionality Through Iterative Learning

5min

19.

Is There a Way to Improve Model Performance?

3min

20.

Is It a Dropout Intuition?

2min

21.

The Later Layers Are Learning of the Tile of All Features

2min

22.

The Fortuitous Forgetting Paper

5min

23.

Is Knowledge Evolution Really Useful in Transfer Learning?

5min

24.

Unsupervised Environment Design

3min

25.

Are You Using Your Model to Identify Desirable Versus Unwanted Information?

4min

26.

Is There a Generalization of Desirable Versus Unwanted?

3min

27.

Getting Rid of Specifics for Spurious Features

2min

28.

Is There a Difference Between a Cow and a Grass Cow?

2min

29.

How Do You Interpret a Scene?

2min

30.

Is There a Way to Unlearn?

5min

31.

Is There a Difference Between Chris Ola and Grande?

3min

32.

Is There a Limit to Context Learning?

3min

33.

Is Your Model Not Reasoning?

3min

34.

Is There a Scalable Compositionality?

5min

35.

Compositional and Trans-Horror Red Scale Part 2

2min

36.

Is There a Culture That Makes You More Effective?

3min

Hattie Zhou is a Ph.D. student at Mila working with Hugo Larochelle and Aaron Courville. Her research focuses on understanding how and why neural networks work, starting with deconstructing why lottery tickets work and most recently exploring how forgetting may be fundamental to learning. Prior to Mila, she was a data scientist at Uber and did research with Uber AI Labs. In this episode, we chat about supermasks and sparsity, coherent gradients, iterative learning, fortuitous forgetting, and much more.

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.