Nested Learning: The Illusion of Deep Learning Architecture

In this episode, we discuss Nested Learning: The Illusion of Deep Learning Architecture by The authors of the paper "Nested Learning: The Illusion of Deep Learning Architecture" are: - Ali Behrouz - Meisam Razaviyayn - Peilin Zhong - Vahab Mirrokni. The paper introduces Nested Learning (NL), a new paradigm framing machine learning as multiple nested optimization problems with distinct context flows, explaining in-context learning in large models. It proposes more expressive optimizers as associative memory modules, a self-modifying sequence model that learns its own update rules, and a continuum memory system to improve continual learning. Together, these contributions enable a continual learning module called Hope, which shows promise in language modeling, knowledge integration, and long-context reasoning tasks.