The Nonlinear Library LW - Dialogue introduction to Singular Learning Theory by Olli Järviniemi
Jul 8, 2024
14:32
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Dialogue introduction to Singular Learning Theory, published by Olli Järviniemi on July 8, 2024 on LessWrong.
Alice: A lot of people are talking about Singular Learning Theory. Do you know what it is?
Bob: I do. (pause) Kind of.
Alice: Well, I don't. Explanation time?
Bob: Uh, I'm not really an expert on it. You know, there's a lot of materials out there that
Alice: that I realistically won't ever actually look at. Or, I've looked at them a little, but I still have basically no idea what's going on. Maybe if I watched a dozen hours of introductory lectures I'd start to understand it, but that's not currently happening.
What I really want is a short overview of what's going on. That's self-contained. And easy to follow. Aimed at a non-expert. And which perfectly answers any questions I might have. So, I thought I'd ask you!
Bob: Sorry, I'm actually really not
Alice: Pleeeease?
[pause]
Bob: Ah, fine, I'll try.
So, you might have heard of ML models being hard to interpret. Singular Learning Theory (SLT) is an approach for understanding models better. Or, that's one motivation, at least.
Alice: And how's this different from a trillion other approaches to understanding AI?
Bob: A core perspective of SLT is studying how the model develops during training. Contrast this to, say, mechanistic interpretability, which mostly looks at the fully trained model. SLT is also more concerned about higher level properties.
As a half-baked analogue, you can imagine two approaches to studying how humans work: You could just open up a human and see what's inside. Or, you could notice that, hey, you have these babies, which grow up into children, go through puberty, et cetera, what's up with that? What are the different stages of development? Where do babies come from? And SLT is more like the second approach.
Alice: This makes sense as a strategy, but I strongly suspect you don't currently know what an LLM's puberty looks like.
Bob: (laughs) No, not yet.
Alice: So what do you actually have?
Bob: The SLT people have some quite solid theory, and some empirical work building on top of that. Maybe I'll start from the theory, and then cover some of the empirical work.
Alice: (nods)
I. Theoretical foundations
Bob: So, as you know, nowadays the big models are trained with gradient descent. As you also know, there's more to AI than gradient descent. And for a moment we'll be looking at the Bayesian setting, not gradient descent.
Alice: Elaborate on "Bayesian setting"?
Bob: Imagine a standard deep learning setup, where you want your neural network to classify images, predict text or whatever. You want to find parameters for your network so that it has good performance. What do you do?
The gradient descent approach is: Randomly initialize the parameters, then slightly tweak them on training examples in the direction of better performance. After a while your model is probably decent.
The Bayesian approach is: Consider all possible settings of the parameters. Assign some prior to them. For each model, check how well they predict the correct labels on some training examples. Perform a Bayesian update on the prior. Then sample a model from the posterior. With lots of data you will probably obtain a decent model.
Alice: Wait, isn't the Bayesian approach very expensive computationally?
Bob: Totally! Or, if your network has 7 parameters, you can pull it off. If it has 7 billion, then no. There are way too many models, we can't do the updating, not even approximately.
Nevertheless, we'll look at the Bayesian setting - it's theoretically much cleaner and easier to analyze. So forget about computational costs for a moment.
Alice: Will the theoretical results also apply to gradient descent and real ML models, or be completely detached from practice?
Bob: (winks)
Alice: You know what, maybe I'll just let you t...
Alice: A lot of people are talking about Singular Learning Theory. Do you know what it is?
Bob: I do. (pause) Kind of.
Alice: Well, I don't. Explanation time?
Bob: Uh, I'm not really an expert on it. You know, there's a lot of materials out there that
Alice: that I realistically won't ever actually look at. Or, I've looked at them a little, but I still have basically no idea what's going on. Maybe if I watched a dozen hours of introductory lectures I'd start to understand it, but that's not currently happening.
What I really want is a short overview of what's going on. That's self-contained. And easy to follow. Aimed at a non-expert. And which perfectly answers any questions I might have. So, I thought I'd ask you!
Bob: Sorry, I'm actually really not
Alice: Pleeeease?
[pause]
Bob: Ah, fine, I'll try.
So, you might have heard of ML models being hard to interpret. Singular Learning Theory (SLT) is an approach for understanding models better. Or, that's one motivation, at least.
Alice: And how's this different from a trillion other approaches to understanding AI?
Bob: A core perspective of SLT is studying how the model develops during training. Contrast this to, say, mechanistic interpretability, which mostly looks at the fully trained model. SLT is also more concerned about higher level properties.
As a half-baked analogue, you can imagine two approaches to studying how humans work: You could just open up a human and see what's inside. Or, you could notice that, hey, you have these babies, which grow up into children, go through puberty, et cetera, what's up with that? What are the different stages of development? Where do babies come from? And SLT is more like the second approach.
Alice: This makes sense as a strategy, but I strongly suspect you don't currently know what an LLM's puberty looks like.
Bob: (laughs) No, not yet.
Alice: So what do you actually have?
Bob: The SLT people have some quite solid theory, and some empirical work building on top of that. Maybe I'll start from the theory, and then cover some of the empirical work.
Alice: (nods)
I. Theoretical foundations
Bob: So, as you know, nowadays the big models are trained with gradient descent. As you also know, there's more to AI than gradient descent. And for a moment we'll be looking at the Bayesian setting, not gradient descent.
Alice: Elaborate on "Bayesian setting"?
Bob: Imagine a standard deep learning setup, where you want your neural network to classify images, predict text or whatever. You want to find parameters for your network so that it has good performance. What do you do?
The gradient descent approach is: Randomly initialize the parameters, then slightly tweak them on training examples in the direction of better performance. After a while your model is probably decent.
The Bayesian approach is: Consider all possible settings of the parameters. Assign some prior to them. For each model, check how well they predict the correct labels on some training examples. Perform a Bayesian update on the prior. Then sample a model from the posterior. With lots of data you will probably obtain a decent model.
Alice: Wait, isn't the Bayesian approach very expensive computationally?
Bob: Totally! Or, if your network has 7 parameters, you can pull it off. If it has 7 billion, then no. There are way too many models, we can't do the updating, not even approximately.
Nevertheless, we'll look at the Bayesian setting - it's theoretically much cleaner and easier to analyze. So forget about computational costs for a moment.
Alice: Will the theoretical results also apply to gradient descent and real ML models, or be completely detached from practice?
Bob: (winks)
Alice: You know what, maybe I'll just let you t...