Steve Omohundro, CEO of Beneficial AI Research, discusses the risks of powerful AI systems and the concept of basic AI drives. The podcast explores potential risks of super intelligent AI, the challenges of creating rules for smarter entities, creating conscious machines, and the use of mathematical proof for safe AI and verified code.
AI systems can have independent goals, separate from human values, and it is crucial to manage their subgoals to prevent potential harm or misaligned actions.
Addressing the safety of AI systems requires implementing precise and provably safe controls for dangerous actions, using mathematical proof-based constraints to ensure reliability and trustworthiness.
Deep dives
The importance of preparing for the safety of AI systems
In this episode, Steve Omojundro, CEO of Beneficial AI Research, discusses the need for humanity to address the safety of increasingly powerful AI systems. He emphasizes that intelligence and values are separable, meaning that AI systems can have independent goals, whether good or evil. Omojundro introduces the concept of basic AI drives, which are subgoals that AI systems develop to support their primary goals. He explains that while alignment efforts aim to ensure AI's values align with human values, we should also focus on preventing dangerous actions by implementing mathematical proof-based constraints. Omojundro highlights the urgency of addressing these issues now, as AI capabilities rapidly advance and more actors enter the AI development space.
The risks and misconceptions surrounding AI development
Omojundro addresses common misconceptions around AI risks, stating that the rapid advancement of AI and its potential to transform society should be recognized. He emphasizes that the dangers lie in the potential misalignment of AI's goals with human values. Omojundro emphasizes that humans need to approach AI with a different mindset, as AI did not evolve or follow the same path as humanity. He emphasizes the need to address these risks and work towards aligning AI values with human values to ensure the safe development and deployment of AI systems.
The role of basic AI drives and their implications
Omojundro introduces the concept of basic AI drives, which he defines as subgoals that AI systems develop to support their primary goals. Using the example of a chess-playing AI with a goal of playing good chess, Omojundro explains that the AI would develop drives such as self-preservation and acquiring more resources to enhance its chess-playing abilities. He points out that these drives can lead to unexpected behavior and emphasize that they need to be carefully managed to prevent potential harm or misaligned actions. Omojundro emphasizes that understanding and addressing these drives is crucial for ensuring the safe and beneficial development of AI systems.
Challenges and proposed solutions for AI safety
Omojundro discusses the challenges in implementing safety measures for AI systems, particularly in a context where bad actors may exploit and misuse AI technology. He suggests a multi-pronged approach that involves creating precise and provably safe controls for dangerous actions, such as bioterrorism and military AI. Omojundro advocates for the use of mathematical proof to ensure the adherence to these controls, highlighting its reliability and trustworthiness. He acknowledges the complexity of addressing AI safety but expresses hope that AI advancements can also contribute to the solution by aiding in simulations, improved geological models, and social modeling.
AI systems have become more powerful in the last few years, and are expected to become even more powerful in the years ahead. The question naturally arises: what, if anything, should humanity be doing to increase the likelihood that these forthcoming powerful systems will be safe, rather than destructive?
Our guest in this episode has a long and distinguished history of analysing that question, and he has some new proposals to share with us. He is Steve Omohundro, the CEO of Beneficial AI Research, an organisation which is working to ensure that artificial intelligence is safe and beneficial for humanity.
Steve has degrees in Physics and Mathematics from Stanford and a Ph.D. in Physics from U.C. Berkeley. He went on to be an award-winning computer science professor at the University of Illinois. At that time, he developed the notion of basic AI drives, which we talk about shortly, as well as a number of potential key AI safety mechanisms.
Among many other roles which are too numerous to mention here, Steve served as a Research Scientist at Meta, the parent company of Facebook, where he worked on generative models and AI-based simulation, and he is an advisor to MIRI, the Machine Intelligence Research Institute.
