Speaker 2
So what big view that I think is actually a misconception of what people worried about AI misalignment have been saying, but I understand why people have this misconception, is people get really fixated on the idea of human values being really complicated and hard to specify and hard to understand. And they're worried about AI systems that are really good at things like
Speaker 1
physics and math and
Speaker 2
but basically just don't get what it is that humans want to see from them, and what human values really are. So
Speaker 2
example that sometimes people bring out is you ask your AI system to cook dinner, your robot to cook
Speaker 2
it doesn't understand that you wouldn't want it to cook the cat if you didn't have any ham in the fridge or something like that.
Speaker 2
kind of worry is something that I think is quite overrated.
Speaker 1
I actually think that, in fact,
Speaker 2
having a basic understanding of human psychology and what humans would think is preferable and not preferable is not a harder problem than understanding physics or understanding how to code and so on. So I expect AI's will perfectly well understand what humans want from them. And so I actually don't expect to see mistakes that seem so egregious as cooking the family's cat for dinner, because the AI systems will understand that humans are going to come home and look at what they did and then determine a reward, like take some action based on that,
Speaker 7
and we'll know that humans
Speaker 2
will be displeased if they come home to see that the cat has been killed and
Speaker 1
cooked. And in fact,
Speaker 2
a lot of my worries sort of stem from the opposite thing, stem from expecting AI systems to have a really good psychological model of humans. And so worrying that will end up in a world where
Speaker 2
to be really getting a lot of subtle nuances and appear to be generalizing really well, while sometimes being deliberately deceptive. Ezra Klein on punctuated equilibrium.
Speaker 8
You need ideas on the shelf, not in your drawer. Don't put it in your drawer. They need to be on a shelf where other people can reach them to shift metaphor a little bit here. You need ideas that are out there. So this is a governing model that in the political science literature is called punctuated equilibrium. Nothing happens and then all of a sudden it does. Puncture in the equilibrium and new things are possible. And or as it's put more commonly, you never let a crisis go to waste. And when there is a crisis, people have to pick up the ideas that are around. And a couple things are important for that. One is that the ideas have to be around. Two is that they have to be coming from a source people trust, right, or have reason to believe they should trust. And three, they have to have some relationship with that source. So what you want to be doing is building relationships with the kinds of people who are going to be, you know, making the decisions. What you want to be doing is building up your own credibility as a source on these issues. And what you want to be doing is actually building up good ideas and battle testing them and getting people to critique them and putting them out in detail, right? I think it is very unlikely that air regulation is going to come out of a less wrong post. But I have seen a lot of good ideas from most wrong posts ending up in, you know, different white paper proposals that now get floated around. And you need a lot more of those. It's funny because, you know, and I've seen this happen in Congress again and again and again, you might wonder, like, why do these think tanks produce all these white papers, you know, reports that truly nobody reads? And there's a panel that nobody's at. It's a lot of work for nobody to read your thing and nobody to come to your speech. But it's not really nobody. It's it. It may really be that only seven people read that report, but five of them were congressional staffers who had to work on this issue. And like, that's what this whole economy is. It is amazing to me. The books that you've never heard of that have ended up hugely influencing national legislation, right? Most people have not read Jump Starting America by John Gruber and Simon Johnson. But as I understand it, it's actually a pretty important part of the chips bill. And so you have to build the ideas. You have to make the ideas legible, incredible to people. And you have to know the people you're trying to make these ideas legible, incredible to. Like, that is like the process by which, you know, you become part of this when it happens.
Speaker 2
Alison Young on the case of the found smallpox files.
Speaker 6
So around the same time the CDC was having all kinds of incidents in 2014. In the middle of all of that, there was a cold storage room on the campus of the National Institutes of Health, just north of Washington, D.C., where they were moving around some old cardboard boxes and they look inside and they see all of these little tiny, very fragile vials from decades ago that are labeled in typewriter print with various pathogens names on them. And it's powdered material. And as they're going through these glass vials, they see some that are labeled as variola.
Speaker 1
Which, yeah, just to be totally clear is smallpox. Exactly. Variola is the pathogen that causes smallpox. Okay, so go on. They found vials of smallpox in a box in a storage room.
Speaker 6
Exactly, in an unlocked storage room. And so this should have been incredibly concerning because smallpox is incredibly deadly. It has been eradicated from the planet. And smallpox virus is only supposed to be found under treaties in two labs in the world. One is in Russia and the other is a specific lab on the campus at the Centers for Disease Control and Prevention in Atlanta. And so these vials shouldn't have been in this cold storage room at NIH. What was also concerning was how they responded to it when they found these files. Ultimately, it was one scientist by themselves who basically picked up the cardboard box and walked it down the corridors of this building at the NIH and across the street into another building. All the while, they're hearing this clink, clink, clink of these fragile old vials hitting each other as they're walking along. The FBI report that I read of the incident criticized the scientist and just the whole handling of this box because when it was properly catalogued in the end, there was a vial that had broken inside this box. And once again, the world got lucky. And it was not smallpox virus. It was some sort of a tissue sample. But as the FBI report noted that had that been the freeze-dried smallpox specimen, there was nothing really protecting the person who was carrying it. It's