The Dangers of Expecting Superhuman AI Systems to Have Goals

Katja Grace is a researcher and writer. She runs AI Impacts, a research project trying to incrementally answer decision-relevant questions about the future of artificial intelligence (AI). Katja blogs primarily at worldspiritsockpuppet, and indirectly at Meteuphoric, Worldly Positions, LessWrong and the EA Forum.

We discuss:

• What is AI Impacts working on?
• Counterarguments to the basic AI x-risk case
• Reasons to doubt that superhuman AI systems will be strongly goal-directed
• Reasons to doubt that if goal-directed superhuman AI systems are built, their goals will be bad by human lights
• Aren't deep learning systems fairly good at understanding our 'true' intentions?
• Reasons to doubt that (misaligned) superhuman AI would overpower humanity
• The case for slowing down AI
• Is AI really an arms race?
• Are there examples from history of valuable technologies being limited or slowed down?
• What does Katja think about the recent open letter on pausing giant AI experiments?
• Why read George Saunders?

Key links:

• World Spirit Sock Puppet (Katja's main blog)
• Counterarguments to the basic AI x-risk case
• Let's think about slowing down AI
• We don't trade with ants
• Thank You, Esther Forbes (George Saunders)

You can see more links and a full transcript at hearthisidea.com/episodes/grace.