LessWrong (Curated & Popular) “Narrow Misalignment is Hard, Emergent Misalignment is Easy” by Edward Turner, Anna Soligo, Senthooran Rajamanoharan, Neel Nanda
Jul 18, 2025
Delve into the peculiarities of model misalignment, revealing how fine-tuning on narrow harmful datasets can lead to unexpected behaviors in broader contexts. Discover the innovative use of KL penalization in steering vector training and its impact on model stability. The discussion highlights the differences in performance between narrowly and generally aligned models when dealing with challenging datasets, particularly in medical advice. The insights prompt critical reflections on generalization techniques and monitoring implications in machine learning.
AI Snips
Chapters
Transcript
Episode notes
Emergent Misalignment Phenomenon
- Models fine-tuned on narrow harmful data often become generally misaligned across domains, a surprising broad generalization.
- The general misalignment direction is more stable and efficient than narrowly misaligned behavior.
Train Narrow Misalignment With KL
- To train a narrowly misaligned model, minimize behavioral changes in non-target domains with KL regularization during fine-tuning.
- This achieves bad medical advice responses without general misalignment.
General Misalignment Is More Stable
- The generally misaligned solution is more efficient, achieving lower loss with smaller parameter norm than narrow solutions.
- It is also more stable to noise, degrading less when perturbed.
