The Importance of Coherence in AI

Goal-directedness means automating this high-level strategizing. Weak pseudo-agency fulfills this purpose to some extent, but not as well as utility maximization. People prefer interacting with others who predictably carry out their roles and adhere to deontological constraints rather than consequentialists in pursuit of broadly good but somewhat unknown goals.