The Evolution of Utility Maximizers in AI

There is a large space of systems which strongly increase the chance of some desirable objective, over-occurring. For instance, without searching out novel ways of making O occur, or modifying themselves to be more consistently O-maximizing, call these weak pseudo-agents. Humans might not be very far on this spectrum, but they seem enough like utility maximizers already to be alarming. It is not clear that economic incentives gradually favor the far end of this spectrum over weak pseudo-agency. There aren't any kind of goal-directedness is incentivized in AI systems.