Understanding Artificial General Intelligence | 16min snip from "The Cognitive Revolution" | AI Builders, Researchers, and Live Player Analysis

Speaker 2
This ties in this other paper that you were involved in writing called Levels of AGI for Operationalizing Progress on the Path to AGI. I think it's a little bit of a mouthful. So we've been referring to that paper as the what is AGI paper, which is a bit more catchy. Yeah, what about the nature of AGI did you want to point to in that paper that you thought like many people were missing?
Speaker 1
That paper primarily was trying to offer a definition and conception of AGI that we thought was fairly implicit in how most people talked about AGI. And we just wanted to provide a rigorous statement of it so that we could move on from this, think, unhelpful strand of dialogue, which said AGI is poorly defined or AGI means different things to everyone. We think, yeah, we've received very positive feedback on that paper. And I think most people think that is a reasonable and useful conception of AGI. In the paper, we discussed some of this, but there's some prior ideas that I would want to call out, which is that, yeah, AGI is a complex concept. It's multidimensional, and it is prone to fallacies of reasoning in people who use it uncritically. So let's unpack that a bit. One fallacy is people think AGI is like human level AI. They think of it as a point, a single system, a single kind of system. And often they think it's like human-like AI. So it has the same kind of strengths and weaknesses of humans. We know that's very unlikely to be the case. it is, historically, AI systems are often much better than us at some things, chess, memory, mathematics, than other things. And so I think we should expect AI systems and AGI to be highly imbalanced in the sense of what they're good at and what they're bad at doesn't look like what we're good at and what we're bad at. Secondly, there's the risk that the concept of AGI leads people to try to build human-like AI. We want to build AI in our image. And some people have argued that's a mistake because that leads to more labor substitution than would otherwise be the case. From this economics point of view, we want to build AI that's as different from us as possible because that's the most complementary to human labor in the economy. If we have like an example is AlphaFold, this Google DeepMind AI system, that's a narrow AI system that's very good at predicting the structure of proteins. It's a great complement to humans because it's not doing what we do. It's not writing emails and, you know, writing strategy memos. It's predicting the structure of proteins, which is not something that humans could do. No one's losing
Speaker 2
their job to alpha fold two. Exactly.
Speaker 1
But it is enabling all kinds of new forms of productivity in medicine and health research that otherwise would not be possible. So arguably, that's the kind of AI systems we should be trying to build. Alien, complementary, some would say narrow AI systems. Another aspect of the concept of AGI is it's arguably pointing us in the direction of general intelligence, which some people would argue is a mistake. We should go towards these systems of narrow AI systems. That's safer. Maybe they would say it's even more likely. General intelligence, they might argue, is not a thing. I'm less convinced by that argument. I do think general intelligence is... There's a kind of implicit claim about the nature of technology, about this implicit logic of technological development. I do think general intelligence is likely to be an important phenomenon that will win out. One will be more willing to employ a general intelligence AI system than a narrow intelligence AI system in more and more domains. We've seen that in the past few years with large language models, that the best poetry language model or the best email writing language model or the best historical language model is often the same language model. It's the one that's trained on the full corpus of human text. And the logic of what's going on there is there's enough spillover in lessons between poetry and math and history and philosophy that your philosophy AI is made better by reading poetry than just having separate, you know, poetry language models and philosophy language models and so forth.
Speaker 2
I mean, it's amazing thing about the structure of knowledge that I guess we didn't necessarily know that before, maybe because I think in humans, it's true that we specialize much more, that it's very hard to be the best poet and the best philosopher and the best chemist. But I guess maybe when your brain can scale as much as you want, you have virtually unlimited ability to read all these different things. And in fact, there are enough connections between them that you can be the best at all of them simultaneously. And
Speaker 1
I think it's an important empirical phenomenon to track. It need not be true that the best physics or philosophy language is also the best poetry or politics or history language model.
Speaker 2
Because you can imagine in the future it might come apart if there's more effort to develop really, really good specialized models that at the frontier, perhaps this isn't the case anymore.
Speaker 1
Yeah. And it certainly seems like a coding language model doesn't need to know history. So even if reading history helped it in some sense know something about coding, maybe later we should distill out most of the historical knowledge so that the model is smaller and more efficient. So this is a phenomenon to track. And I think implicit to the notion of AGI is that general intelligence will be important. We're not kind of at peak returns to generality. There will continue to be returns. So we'll continue to see these large models being trained.
Speaker 2
Are there any other pros or cons of the generality that are worth flagging?
Speaker 1
Yeah, and the concept of AGI. So I think one thing that really makes the concept of AGI useful, and a lot of people critique it or say they don't want to use the concept. I've rarely seen an alternative for what we're trying to point to. Sometimes people say transformative AI. I don't think that's adequate. Transformative AI is usually defined as AI systems that have some level of impact on the economy, on like at the scale of the industrial revolution. And the problem with that is you can get a transformative impact from a narrow AI system. So you could have an AI system that poses a narrow catastrophic risk that is not general intelligence. I do think there's something important to name about AGI.
Speaker 2
I think you often see this phenomenon where people will say, this term is kind of confused, it's too much of a cluster of different competing things, but despite all the criticisms and worries, people just completely keep using it all the time and they always come back to it. And I think at that point, you just have to concede there's actually something super important here that people are desperate to refer to. And we just have to clarify this idea rather than give up on it.
Speaker 1
Yeah. And yeah. And it's good that we always try to refine our conceptual toolbox. And there are these other concepts and targets that are worth calling out. So one that's been named by Ajay Akotra and many others is this idea of machine learning, AI that can do machine learning R&D, that can have this kind of recursive self-improvement that need not be general. It could be a kind of narrow AI system, but if it does kick off this recursive process, that is an important phenomenon to, and I think Holden Karnowski has also really drawn attention to this. But yeah, back to general intelligence, I do think AGI is a probable phenomenon. It's probable we'll get general intelligence around the time we get AI systems that can do, you know, radically recursive self-improvement or a lot of other things. On
Speaker 2
that point, I think actually a slightly contrarian take that I have is that people, I think ML people hate this, but the idea that in fact, machine learning research, even the cutting edge stuff, like might actually not be that difficult. That in fact, the kind of things that, if you try to break it down into the specific process by which we're improving these models, you've got like theory generation stage, then you've got, well, how do we actually test this? How do we develop a benchmark? And then you actually run the thing, very compute intensive, and then you go and you decide whether it was an improvement and then go back to the theory generation stage. This might be possible well short of a full AGI that has all these different skills, especially if you focused on it. In fact, maybe a lot of this research is much less difficult in some sense than people who are involved in it might want to believe, and they could be relatively easily automated, which would be quite
Speaker 1
shocking and quite consequential if true. And an interesting phenomenon we've seen in public surveys of AI and ML experts is that they often put automation of the ML process to be very late, one of the last or the last task that's performed by AI systems, and often significantly later than AGI, human level AI, which has been defined in different ways. But even given a quite high definition, the automation of MLR and D is often reported to be later than that. One argument is that this reflects this kind of bias that everyone thinks that their career, the task that they're good at is really hard and special and won't be automated. Maybe. I think there's also a case in favor of the idea that of the set of human tasks, the sort of last thing to be fully automated is the process of improving MLR&D. So we can reflect on that. Coming back to AGI, it does seem useful to point to this space of AI systems. And let's recall, it's a space. It's not a single kind of AI system. Defined as an AI system that's better than humans at most tasks. And there's different ways you can define most and different ways you can define the set of tasks. But given a, so you can say economically relevant tasks, and most could be 99% or 50%, or I think it's usually quite helpful to choose a lot. Because there might be some tail tasks that, for whatever reason, take a long time to automate, and most of the impact will occur before. And then you can also, there's a parameter of how much it means to be better then. So is it better than the median human who's unskilled? I think typically you want to look at skilled humans in that task, because that's the economically relevant threshold. And so this part of the sort of future space that we're looking at, I think, is important to conceptualize because this is the point when labor substitution really takes place. So that has profound economic impacts and political impacts because people are no longer in the role. There's this sort of natural human in the loop that takes place when humans are doing the task is no longer the case. There's also this, I think I call it a performance threshold that we cross when whatever was previously possible with humans, that set is no longer the kind of bounding set of what's possible. Because when the AI is better than humans at some task, there are new qualitatively new things that become possible. This is like AlphaFold. And that also represents an important moment in history when new technologies come online.
Speaker 2
So a reason that this connects is that I think the paper alludes to this idea that as you're approaching AGI, you could start with it being very strong in some areas and relatively weak in other dimensions. And then you've got like an area where it's going to be already like vastly superhuman. And then there's like the last few pieces that come into place where it begins to approach human level. And you could potentially try to change or just choose which ones those are going to be. You know, is it going to be strongest on the cooperative AI stuff to like already? And then we like add in, I don't know, I guess like the technical knowledge or just like practical know-how or agency and so on. Or do we start with the agency and having an enormous of factual knowledge, and then we add in the cooperative stuff later? Do you think that's an important, maybe underrated idea still? And do you have preferences maybe on what things we want to add to the AGI early versus late? Yeah.
Speaker 1
So I think this is a very useful conceptual heuristic, is to think AGI is the space and there's multiple paths that we could take to get to AGI. Coming back to this discussion of technological determinism, there may be, yeah, these different trajectories and it really matters not just, well, it matters what kind of AGI we build. AGI is not a single kind of system. It's a vast set of systems. It's, you know, you can call it the corner of this high-dimensional intelligence space. And so this corner has many different elements. And it matters also how we get there, what the trajectory. As you were saying, we might characterize two sort of character trajectories. One where AI is relatively not that capable at, say, physics or material science, but very good at cooperative skill. And another world where it's superhuman at material science, but amateur at cooperative skill. And we can ask ourselves, which world is safer, more beneficial? Some in the safety community prefer the latter. The story there is we'll unlock these economic advances, health advances, while having AI systems that are socially, strategically simplistic. And so it's easier for humans to manage those AI systems. They're not going to outwit us. Whereas the cooperative AI bet takes the opposite tack. It says a world with accelerating technological advances, capabilities advances, will generate lots of benefits, but also disruptions that we may not be able to adapt to and manage in the time, the rapid rate that they're coming online. Whereas we still have these cooperation problems that we foremost need to solve. so we'd be better off if we sort of take the bet on making AI systems more cooperatively skilled than they otherwise would
Speaker 2
be. I mean, if this is contested by, there's a lot of disagreement between people who are super informed about this. Is it something that we should focus in a lot more and try to reach agreement? Or maybe is it just something where it's just going to be unclear basically up until the day when we have to decide? I
Speaker 1
think it would be, yeah, it's an important part of the cooperative AI research agenda to articulate this argument and to host the debate and to try and make sense of it. You know, you don't want to invest too much of your time and resources going down a path that you're not sure is beneficial. And I think that's true of also, you know, all these sort of pro-social bets. It is worth investing a significant share of our effort there, making sure the bet is beneficial. So
Speaker 2
where can people go to learn more about this sort of debate? I think it's the Machine Intelligence Research Institute folks who are a bit more wary of the high social skill, high cooperation early agenda, right? And I guess I'm not sure who's advocating for, I guess you and your co-authors are advocating for cooperative AI in particular. Yeah.
Speaker 1
So for cooperative AI, I'd point them to the Cooperative AI Foundation and reach out to anyone there or me and we'll invite you to the conversation. And yeah, on the contrary point, I would say Eliezer Yudkowsky, Nate Soares have expressed this view fairly strongly in the past. So they might be interested in continuing to argue that position, but I can't speak for them. And then I would also say on this kind of third poll, Paul Cristiano and Carl Schulman, I think have articulated the case for this super cooperative AGI hypothesis. This is that it is very important that AGI systems be able to cooperate with each other. So in that sense, they agree with the cooperative AI bet, but they think it will come as a byproduct of AGI. And so they don't think it's necessarily an area we need to invest in. All
Speaker 2
right.

Agency over AI? Allan Dafoe on Technological Determinism & DeepMind's Safety Plans, from 80000 Hours

"The Cognitive Revolution" | AI Builders, Researchers, and Live Player Analysis

Remember Everything You Learn from Podcasts