How we form and revise intuitions

Related to: Commonsense Good, Creative Good (and my comment); Ethical Injunctions.

Epistemic status: I’m fairly sure “ethics” does useful work in building human structures that work. My current explanations of how are wordy and not maximally coherent; I hope you guys help me with that.

Introduction

It is intractable to write large, good software applications via spaghetti code – but it's comparatively tractable using design patterns (plus coding style, attention to good/bad codesmell, etc.).

I’ll argue it is similarly intractable to have predictably positive effects on large-scale human stuff if you try it via straight consequentialism – but it is comparatively tractable if you use ethical heuristics, which I’ll call “ethical design patterns,” to create situations that are easier to reason about. Many of these heuristics are honed by long tradition (eg “tell the truth”; “be kind”), but sometimes people successfully craft new “ethical design patterns” fitted to a [...]

---

Outline:

(00:31) Introduction

(01:32) Intuitions and ground truth in math, physics, coding

(02:08) We revise our intuitions to match the world. Via deliberate work.

(03:08) We design our built world to be intuitively accessible

(04:22) Intuitions and ground truth in ethics

(04:52) We revise our ethical intuitions to predict which actions we'll be glad of, long-term

(06:27) Ethics helps us build navigable human contexts

(09:30) We use ethical design patterns to create institutions that can stay true to a purpose

(12:17) Ethics as a pattern language for aligning mesaoptimizers

(13:08) Examples: several successfully crafted ethical heuristics, and several gaps

(13:15) Example of a well-crafted ethical heuristic: Don't drink and drive

(14:45) Example of well-crafted ethical heuristic: Earning to give

(15:10) A partial example: YIMBY

(16:24) A historical example of gap in folks' ethical heuristics: Handwashing and childbed fever

(19:46) A contemporary example of inadequate ethical heuristics: Public discussion of group differences

(25:04) Gaps in our current ethical heuristics around AI development

(26:30) Existing progress

(28:30) Where we still need progress

(32:21) Can we just ignore the less-important heuristics, in favor of 'don't die'?

(35:02) These gaps are in principle bridgeable

(36:29) Related, easier work

The original text contained 12 footnotes which were omitted from this narration.

---

First published:
September 30th, 2025

Source:
https://www.lesswrong.com/posts/E9CyhJWBjzoXritRJ/ethical-design-patterns-1

---



Narrated by TYPE III AUDIO.