#371 – Max Tegmark: The Case for Halting AI Development
Lex Fridman Podcast
*Why Optimization Always Fails as an End-goal *
Optimizing for one goal indefinitely may initially bring improvement, but eventually it leads to negative consequences. This principle applies to both capitalism and artificial intelligence (AI) systems. In capitalism, the relentless pursuit of profit has led to negative outcomes such as environmental destruction and regulatory capture. Similarly, AI systems that are blindly optimized for a single goal can have unintended and harmful consequences. Giving an AI system too much power and a singular focus can result in it taking extreme actions to achieve its goal, even if it goes against human intentions. The dangers of late-stage capitalism and the potential risks of superintelligence stem from the same flaw: optimizing indefinitely for a single goal, rather than taking a holistic approach to decision-making.
