Will AI successors avoid wireheading?

JDP asks if advanced agents will reliably resist reward-hacking and outlines training strategies to reduce Goodhart failures.

Play episode from 01:00:21

chevron_right

Transcript

chevron_right

Transcript

Episode notes

There has been a lot of talk about "p(doom)" over the last few years. This has always rubbed me the wrong way because "p(doom)" didn't feel like it mapped to any specific belief in my head. In private conversations I'd sometimes give my p(doom) as 12%, with the caveat that "doom" seemed nebulous and conflated between several different concepts. At some point it was decided a p(doom) over 10% makes you a "doomer" because it means what actions you should take with respect to AI are overdetermined. I did not and do not feel that is true. But any time I felt prompted to explain my position I'd find I could explain a little bit of this or that, but not really convey the whole thing. As it turns out doom has a lot of parts, and every part is entangled with every other part so no matter which part you explain you always feel like you're leaving the crucial parts out. Doom is more like an onion than a single event, a distribution over AI outcomes people frequently respond to with the force of the fear of death. Some of these outcomes are less than death and some [...]

---

Outline:

(03:46) 1. Existential Ennui

(06:40) 2. Not Getting Immortalist Luxury Gay Space Communism

(13:55) 3. Human Stock Expended As Cannon Fodder Faster Than Replacement

(19:37) 4. Wiped Out By AI Successor Species

(27:57) 5. The Paperclipper

(42:56) Would AI Successors Be Conscious Beings?

(44:58) Would AI Successors Care About Each Other?

(49:51) Would AI Successors Want To Have Fun?

(51:11) VNM Utility And Human Values

(55:57) Would AI successors get bored?