If you have a system where there is no particular thing likely to the function, then the system is still going to produce various actions that might be very instrumentally powerful. It's ironic you mentioned utility function since in a recent episode with Vernon Smith, we talked about how the utility function approach to the theory of the consumer is somewhat limiting. But the part that I'm hard for me to understand is that, let's talk about deep blue, the computer that plays chess and now we understand that computers play chess better than humans. That's all it does.
Nick Bostrom of the University of Oxford talks with EconTalk host Russ Roberts about his book, Superintelligence: Paths, Dangers, Strategies. Bostrom argues that when machines exist which dwarf human intelligence they will threaten human existence unless steps are taken now to reduce the risk. The conversation covers the likelihood of the worst scenarios, strategies that might be used to reduce the risk and the implications for labor markets, and human flourishing in a world of superintelligent machines.