LessWrong (Curated & Popular)

“The Problem with Defining an ‘AGI Ban’ by Outcome (a lawyer’s take).” by Katalina Hernandez

Sep 21, 2025
Katalina Hernandez, a practicing lawyer and expert in AGI policy, dives deep into the complexities of regulating artificial general intelligence. She explains why defining AGI based on potential outcomes, like human extinction, is legally inadequate. Instead, she argues for precise, enforceable definitions that focus on precursor capabilities such as autonomy and deception. Citing lessons from nuclear treaties, Katalina emphasizes the importance of establishing bright lines to enable effective regulation and prevent disastrous risks.
Ask episode
AI Snips
Chapters
Transcript
Episode notes
INSIGHT

Outcome Definitions Fail Legally

  • Defining AGI by its catastrophic outcome (e.g., human extinction) is legally useless because regulation must act before harm occurs.
  • Focus regulation on capabilities and precursors that make catastrophe possible, not on the final outcome.
ADVICE

Enact Strict Liability For Precursors

  • Do legislate strict liability for risky precursor states so enforcement can occur before catastrophe.
  • Ban precursor capabilities that causally enable extinction, not just bad outcomes.
INSIGHT

Fines Enable Goodharting Risks

  • Civil product-safety style bans with fines encourage companies to Goodhart compliance rather than prevent risky development.
  • Monetary penalties alone won't stop firms from legally pushing toward dangerous systems.
Get the Snipd Podcast app to discover more snips from this episode
Get the app