LessWrong (30+ Karma) “Please, Don’t Roll Your Own Metaethics” by Wei Dai
Nov 12, 2025
In this discussion, Wei Dai, a cryptography expert and applied philosophy writer, draws intriguing parallels between cryptography and metaethics. He shares a memorable experience from his internship, highlighting the pitfalls of trusting amateur-designed systems. Dai passionately argues against creating bespoke metaethics, emphasizing the inherent dangers and overconfidence in novel philosophical positions. He stresses the need for humility in discussions about AI risks while inviting feedback to improve understanding in this complex field.
AI Snips
Chapters
Transcript
Episode notes
Breaking A Homegrown PRNG
- Wei Dai recounts breaking an amateur pseudorandom number generator during an internship to convince its proposer to use established crypto.
- He used no prior symmetric cryptanalysis experience yet found a clean attack that persuaded the inventor to abandon the design.
Crypto Has Clear Failure Modes
- Cryptographers quickly learn how easy it is to be overconfident in one's own ideas and internalize "don't roll your own crypto."
- Applied philosophy lacks similarly clear tests and standard libraries, so overconfidence is harder to correct.
Philosophy Lacks Decisive Tests
- In philosophy and AI alignment it's rarely possible to produce clean, decisive attacks that prove ideas wrong.
- That makes persuading others or establishing standardized fallback methods much harder than in cryptography.