Exploring Developmental Interpretability in Learning Models

What's going on with deep learning? What sorts of models get learned, and what are the learning dynamics? Singular learning theory is a theory of Bayesian statistics broad enough in scope to encompass deep neural networks that may help answer these questions. In this episode, I speak with Daniel Murfet about this research program and what it tells us.

Patreon: patreon.com/axrpodcast

Ko-fi: ko-fi.com/axrpodcast

Topics we discuss, and timestamps:

0:00:26 - What is singular learning theory?

0:16:00 - Phase transitions

0:35:12 - Estimating the local learning coefficient

0:44:37 - Singular learning theory and generalization

1:00:39 - Singular learning theory vs other deep learning theory

1:17:06 - How singular learning theory hit AI alignment

1:33:12 - Payoffs of singular learning theory for AI alignment

1:59:36 - Does singular learning theory advance AI capabilities?

2:13:02 - Open problems in singular learning theory for AI alignment

2:20:53 - What is the singular fluctuation?

2:25:33 - How geometry relates to information

2:30:13 - Following Daniel Murfet's work

The transcript: https://axrp.net/episode/2024/05/07/episode-31-singular-learning-theory-dan-murfet.html

Daniel Murfet's twitter/X account: https://twitter.com/danielmurfet

Developmental interpretability website: https://devinterp.com

Developmental interpretability YouTube channel: https://www.youtube.com/@Devinterp

Main research discussed in this episode:

- Developmental Landscape of In-Context Learning: https://arxiv.org/abs/2402.02364

- Estimating the Local Learning Coefficient at Scale: https://arxiv.org/abs/2402.03698

- Simple versus Short: Higher-order degeneracy and error-correction: https://www.lesswrong.com/posts/nWRj6Ey8e5siAEXbK/simple-versus-short-higher-order-degeneracy-and-error-1

Other links:

- Algebraic Geometry and Statistical Learning Theory (the grey book): https://www.cambridge.org/core/books/algebraic-geometry-and-statistical-learning-theory/9C8FD1BDC817E2FC79117C7F41544A3A

- Mathematical Theory of Bayesian Statistics (the green book): https://www.routledge.com/Mathematical-Theory-of-Bayesian-Statistics/Watanabe/p/book/9780367734817 In-context learning and induction heads: https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html

- Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity: https://arxiv.org/abs/2106.15933

- A mathematical theory of semantic development in deep neural networks: https://www.pnas.org/doi/abs/10.1073/pnas.1820226116

- Consideration on the Learning Efficiency Of Multiple-Layered Neural Networks with Linear Units: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4404877

- Neural Tangent Kernel: Convergence and Generalization in Neural Networks: https://arxiv.org/abs/1806.07572

- The Interpolating Information Criterion for Overparameterized Models: https://arxiv.org/abs/2307.07785

- Feature Learning in Infinite-Width Neural Networks: https://arxiv.org/abs/2011.14522

- A central AI alignment problem: capabilities generalization, and the sharp left turn: https://www.lesswrong.com/posts/GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization

- Quantifying degeneracy in singular models via the learning coefficient: https://arxiv.org/abs/2308.12108

Episode art by Hamish Doodles: hamishdoodles.com