Dario Amodei’s September 12 essay, We Must Pace the Frontier, set off an unusual public fight. The Anthropic CEO argued that AI capabilities are beginning to advance faster than our ability to understand and control them, and proposed independent evaluators, coordination among frontier labs in democratic countries, and eventually agreements with China.

Underneath that argument sits a harder problem. Amodei repeatedly talks about improving “alignment,” the effort to make powerful AI systems behave according to human intentions and values. Anthropic even describes principles embedded in Claude’s Constitution. But the more capable the intelligence becomes, the harder the obvious question is to avoid: whose values are we aligning it to?

Humans do not have one moral operating system. Values differ across nations, religions, political systems, generations, cultures, communities, and geography. Historical experience changes what people mean by fairness, freedom, security, family, justice, and individual rights. Even within one country, people can disagree fiercely about which of those principles should prevail when they collide.

Perhaps an ASI could be given a thin constitution that sits above those differences. Protect human life. Do not destroy the planet. Do not deliberately cause human extinction. Preserve human autonomy. Those sound close to universal until the system studies us. Humans knowingly kill other humans in wars and self-defense. Governments make decisions that predictably cost lives to protect other interests. Doctors sometimes choose which patient receives a scarce organ. We knowingly damage ecosystems because billions of people depend on the economic activity causing the damage. We routinely violate the clean versions of the principles we would presumably give the machine.

An intelligence vastly smarter than us would see those contradictions immediately. Tell it, “Never harm a human,” and reality will eventually produce situations in which some harm cannot be avoided. Tell it to learn from human behavior, and it may conclude that our supposedly sacred rules contain thousands of accepted exceptions. Tell it to follow our stated values instead, and it may become more faithful to those values than the humans who wrote them.

The alternative is equally strange. Maybe there is no single human-aligned ASI. America develops systems shaped by American laws and norms. China develops systems reflecting Chinese institutions and priorities. Other nations, cultures, religions, and corporations build their own. Instead of one superintelligence aligned with humanity, we get competing superintelligences aligned with different versions of humanity.

At that point, the differences are not confined to how a chatbot answers a controversial question. These systems could be discovering medicines, managing infrastructure, directing economies, conducting scientific research, advising governments, and making decisions whose consequences cross borders. The moral rules inside one system inevitably collide with the moral rules inside another.

The Conundrum:

Do we try to create a basic human constitution that every ASI must follow, accepting that someone must decide which values qualify as universal and how those rules apply when humanity itself routinely violates them?

Or do we allow different societies to align their own ASIs to their own values, preserving cultural and political self-determination while creating a world of superintelligences operating under incompatible definitions of what is right?

A single constitution risks placing humanity under moral rules billions of people never agreed to. Many constitutions risk turning our deepest disagreements into competing intelligences with powers far beyond our own.

What does it actually mean to build an ASI “aligned with humanity” when humanity has never been aligned with itself?

First 24 Hours of a Bioweapon Attack - Annie Jacobsen

Future of Life Institute Podcast

Mother Nature and zoonotic threats

The AI-powered Podcast Player