Active Dendrites avoid catastrophic forgetting - Interview with the Authors

#multitasklearning #biology #neuralnetworks

This is an interview with the paper's authors: Abhiram Iyer, Karan Grewal, and Akash Velu!

Paper Review Video: https://youtu.be/O_dJ31T01i8

Check out Zak's course on Graph Neural Networks (discount with this link): https://www.graphneuralnets.com/p/int...

Catastrophic forgetting is a big problem in mutli-task and continual learning. Gradients of different objectives tend to conflict, and new tasks tend to override past knowledge. In biological neural networks, each neuron carries a complex network of dendrites that mitigate such forgetting by recognizing the context of an input signal. This paper introduces Active Dendrites, which carries over the principle of context-sensitive gating by dendrites into the deep learning world. Various experiments show the benefit in combatting catastrophic forgetting, while preserving sparsity and limited parameter counts.

OUTLINE:

0:00 - Intro

0:55 - Sponsor: GNN Course

2:30 - How did the idea come to be?

7:05 - What roles do the different parts of the method play?

8:50 - What was missing in the paper review?

10:35 - Are biological concepts viable if we still have backprop?

11:50 - How many dendrites are necessary?

14:10 - Why is there a plateau in the sparsity plot?

20:50 - How does task difficulty play into the algorithm?

24:10 - Why are there different setups in the experiments?

30:00 - Is there a place for unsupervised pre-training?

32:50 - How can we apply the online prototyping to more difficult tasks?

37:00 - What did not work out during the project?

41:30 - How do you debug a project like this?

47:10 - How is this related to other architectures?

51:10 - What other things from neuroscience are to be included?

55:50 - Don't miss the awesome ending :)

Paper: https://arxiv.org/abs/2201.00042

Blog: https://numenta.com/blog/2021/11/08/c...

Link to the GNN course (with discount): https://www.graphneuralnets.com/p/int...

Authors: Abhiram Iyer, Karan Grewal, Akash Velu, Lucas Oliveira Souza, Jeremy Forest, Subutai Ahmad

Links:

TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick

YouTube: https://www.youtube.com/c/yannickilcher

Twitter: https://twitter.com/ykilcher

Discord: https://discord.gg/4H8xxDF

BitChute: https://www.bitchute.com/channel/yann...

LinkedIn: https://www.linkedin.com/in/ykilcher