The Nonlinear Library: LessWrong LW - minutes from a human-alignment meeting by bhauth
May 24, 2024
03:41
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: minutes from a human-alignment meeting, published by bhauth on May 24, 2024 on LessWrong.
"OK, let's get this meeting started. We're all responsible for development of this new advanced intelligence 'John'. We want John to have some kids with our genes, instead of just doing stuff like philosophy or building model trains, and this meeting is to discuss how we can ensure John tries to do that."
"It's just a reinforcement learning problem, isn't it? We want kids to happen, so provide positive reinforcement when that happens."
"How do we make sure the kids are ours?"
"There's a more fundamental problem than that: without intervention earlier on, that positive reinforcement will never happen."
"OK, so we need some guidance earlier on. Any suggestions?"
"To start, having other people around is necessary. How about some negative reinforcement if there are no other humans around for some period of time?"
"That's a good one, also helps with some other things. Let's do that."
"Obviously sex is a key step in producing children. So we can do positive reinforcement there."
"That's good, but wait, how do we tell if that's what's actually happening?"
"We have access to internal representation states. Surely we can monitor those to determine the situation."
"Yeah, we can monitor the representation of vision, instead of something more abstract and harder to understand."
"What if John creates a fictional internal representation of naked women, and manages to direct the monitoring system to that instead?"
"I don't think that's plausible, but just in case, we can add some redundant measures. A heuristic blend usually gives better results, anyway."
"How about monitoring the level of some association between some representation of the current situation and sex?"
"That could work, but how do we determine that association? We'd be working with limited data there, and we don't want to end up with associations to random irrelevant things, like specific types of shoes or stylized drawings of ponies."
"Those are weird examples, but whatever. We can just rely on indicators of social consensus, and then blend those with personal experiences to the extent they're available."
"I've said this before, but this whole approach isn't workable. To keep a John-level intelligence aligned, we need another John-level intelligence."
"Oh, here we go again. So, how do you expect to do that?"
"I actually have a proposal: we have John follow cultural norms around having children. We can presume that a society that exists would probably have a culture conducive to that."
"Why would you expect that to be any more stable than John as an individual? All that accomplishes is some averaging, and it adds the disadvantages of relying on communication."
"I don't have a problem with the proposal of following cultural norms, but I think that such a culture will only be stable to the extent that the other alignment approaches we discussed are successful. So it's not a replacement, it's more of a complement."
"We were already planning for some cultural norm following. Anyone opposed to just applying the standard amount of that to sex-related things?"
"Seems good to me."
"I have another concern. I think the effectiveness of the monitoring systems we discussed is going to depend on the amount of recursive self-improvement that happens, so we should limit that."
"I think that's a silly concern and a huge disadvantage. Absolutely not."
"I'm not concerned about the alignment impact if John is already doing some RSI, but we do have a limited amount of time before those RSI investments need to start paying off. I vote we limit the RSI extent based on things like available food resources and life expectancy."
"I don't think everyone will reach a consensus on this issue, so let's just compromise on the amount and metrics."
"Fine."
"A...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: minutes from a human-alignment meeting, published by bhauth on May 24, 2024 on LessWrong.
"OK, let's get this meeting started. We're all responsible for development of this new advanced intelligence 'John'. We want John to have some kids with our genes, instead of just doing stuff like philosophy or building model trains, and this meeting is to discuss how we can ensure John tries to do that."
"It's just a reinforcement learning problem, isn't it? We want kids to happen, so provide positive reinforcement when that happens."
"How do we make sure the kids are ours?"
"There's a more fundamental problem than that: without intervention earlier on, that positive reinforcement will never happen."
"OK, so we need some guidance earlier on. Any suggestions?"
"To start, having other people around is necessary. How about some negative reinforcement if there are no other humans around for some period of time?"
"That's a good one, also helps with some other things. Let's do that."
"Obviously sex is a key step in producing children. So we can do positive reinforcement there."
"That's good, but wait, how do we tell if that's what's actually happening?"
"We have access to internal representation states. Surely we can monitor those to determine the situation."
"Yeah, we can monitor the representation of vision, instead of something more abstract and harder to understand."
"What if John creates a fictional internal representation of naked women, and manages to direct the monitoring system to that instead?"
"I don't think that's plausible, but just in case, we can add some redundant measures. A heuristic blend usually gives better results, anyway."
"How about monitoring the level of some association between some representation of the current situation and sex?"
"That could work, but how do we determine that association? We'd be working with limited data there, and we don't want to end up with associations to random irrelevant things, like specific types of shoes or stylized drawings of ponies."
"Those are weird examples, but whatever. We can just rely on indicators of social consensus, and then blend those with personal experiences to the extent they're available."
"I've said this before, but this whole approach isn't workable. To keep a John-level intelligence aligned, we need another John-level intelligence."
"Oh, here we go again. So, how do you expect to do that?"
"I actually have a proposal: we have John follow cultural norms around having children. We can presume that a society that exists would probably have a culture conducive to that."
"Why would you expect that to be any more stable than John as an individual? All that accomplishes is some averaging, and it adds the disadvantages of relying on communication."
"I don't have a problem with the proposal of following cultural norms, but I think that such a culture will only be stable to the extent that the other alignment approaches we discussed are successful. So it's not a replacement, it's more of a complement."
"We were already planning for some cultural norm following. Anyone opposed to just applying the standard amount of that to sex-related things?"
"Seems good to me."
"I have another concern. I think the effectiveness of the monitoring systems we discussed is going to depend on the amount of recursive self-improvement that happens, so we should limit that."
"I think that's a silly concern and a huge disadvantage. Absolutely not."
"I'm not concerned about the alignment impact if John is already doing some RSI, but we do have a limited amount of time before those RSI investments need to start paying off. I vote we limit the RSI extent based on things like available food resources and life expectancy."
"I don't think everyone will reach a consensus on this issue, so let's just compromise on the amount and metrics."
"Fine."
"A...