LessWrong (Curated & Popular) cover image

"AGI Ruin: A List of Lethalities" by Eliezer Yudkowsky

LessWrong (Curated & Popular)

00:00

Using the Paradime of Loss Functions to Optimize a Cognitive System Will Kill You

There is no known way to use the paradime of loss functions sensory imports and reward imports to optimize anything within a cognitive system. It just isn't true that we know a function on web cam impurt such that every world with that web cam showing the right things is safe for us creatures outside the web cam. Possibilities exist in some possible environments, underlying every given sense impurt. Human raters make systematic errors, regular, compactly describable, predictable errors. All of these kill you if optimized over by a sufficiently powerful intelligence. This general problem is a fact about the territory, not the map. It's a fact aboutThe actual environment, not the particular optimize that lethal to

Play episode from 32:58
Transcript

The AI-powered Podcast Player

Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!
App store bannerPlay store banner
Get the app