80,000 Hours Podcast
#176 – Nathan Labenz on the final push for AGI, understanding OpenAI's leadership drama, and red-teaming frontier models
Episode guests
Podcast summary created with Snipd AI
Quick takeaways
- OpenAI should strengthen control measures for advanced AI models.
- There is a need for understanding and governing advanced AI technology.
- Transparency and collaboration are crucial in the AI community.
- OpenAI should re-evaluate their pursuit of AGI considering control measures.
- OpenAI should provide better understanding of future model predictions.
- OpenAI's focus on responsible and ethical AI development is commendable.
Deep dives
Concerns over OpenAI's control measures for advanced AI
During the podcast episode, concerns were raised about OpenAI's control measures for their advanced AI models. The speaker expressed worry about the rapidly improving capabilities of the models compared to the seemingly inadequate control measures in place. They highlighted their own experience as a red team member and their observations of the models' behavior, including its lack of refusal for unsafe prompts in the safety edition. The speaker also mentioned their attempts to communicate their concerns to OpenAI's board members and their subsequent removal from the red team project. Overall, the speaker emphasized the need for stronger control measures and a better alignment between the power of AI models and their safety considerations.
The need for better understanding and governance of advanced AI
The podcast episode emphasized the importance of understanding and governing advanced AI technology. The speaker discussed the need for organizations and individuals to stay updated on the rapid advancements in AI capabilities. They highlighted the potential impact of AI on various aspects, such as economic growth, job displacement, and AI ethics. The speaker also stressed the significance of having credible sources of information to provide accurate and comprehensive insights about AI developments. They called for broader efforts to evaluate the safety and control measures of AI models, emphasizing the need to address potential risks and ensure responsible AI advancements.
The urgency for transparency and collaboration in the AI community
Transparency and collaboration were highlighted as crucial elements in the AI community. The speaker expressed concern about the lack of information and transparency regarding OpenAI's plans and control measures. They emphasized the need for open dialogue, discussions, and information sharing among developers, researchers, and regulators to address the challenges and uncertainties associated with advanced AI. The speaker called for collective efforts to understand AI technologies better, devise appropriate safety measures, and prevent potential risks. They encouraged the AI community to prioritize responsible AI development and work together to ensure a safer and more informed approach to advancing the technology.
Re-examining the Quest for AGI
The author suggests that OpenAI should re-evaluate their single-minded pursuit of AGI and consider the current capabilities and control measures. With the rapid advancement of capabilities and the slower progress in control measures, it may be wise to question the goal of creating a superhuman AI without clear means of control. The author highlights the divergence between capabilities and controls and urges OpenAI to stay in the current 'base camp' of human-level performance, focusing on useful products rather than the all-encompassing AGI vision.
Better Predictions and Transparency
The author notes the importance of OpenAI providing a better understanding of what they can predict about the capabilities of future models. They suggest that more transparency about the prediction accuracy and the factors that influence performance would be helpful. By sharing insights into the success or shortcomings of their predictions, OpenAI can provide a clearer picture of what to expect from future models, improving trust and understanding among stakeholders and the wider community.
Questioning the 'China Race' Narrative
The author praises OpenAI and specifically Sam Altman for challenging the assumption that the race with China should dictate AGI development. By emphasizing the need for individual decision-making and not blindly following China's actions, OpenAI demonstrates a thoughtful and independent approach. The author encourages OpenAI to continue questioning this narrative and maintain its focus on responsible and ethical AI development, regardless of external pressures.
The Mystery of OpenAI's Board Decision
The podcast discusses the mystery surrounding OpenAI's board decision to remove Sam Altman as the CEO. Despite widespread speculation, the board has not provided a clear explanation for their actions. One possible theory is that there were trust issues between Sam and the board, possibly due to a breakdown in communication or concerns about his consistency. Another theory suggests that a recent breakthrough in capabilities may have influenced the decision. However, the lack of transparency from the board has left many questions unanswered. While speculating about the situation, it is important to remain open-minded and wait for more information to unfold.
The Need for Transparency and Communication
One of the key issues in the OpenAI board's decision to remove Sam Altman was their failure to provide a clear explanation for their actions. This lack of transparency has created confusion and speculation among the public. It is important for the board to communicate their motivations and actions to foster trust and allow people to understand their reasoning. Without a clear explanation, it becomes challenging to fully comprehend the decision and its implications.
The Complexity of OpenAI's Internal Dynamics
The decision to remove Sam Altman as CEO of OpenAI was likely influenced by a combination of factors, including trust issues and concerns about consistency. The board's decision-making process may have been shaped by internal dynamics and power struggles within the organization. It is possible that the board's perception of Sam's leadership and communication style played a role in their decision. However, the precise details remain unknown, and it is essential to await further information before drawing definitive conclusions.
The growing accessibility of fine-tuning models and the need for caution
The release of the llama2 model by Meta (formerly Facebook) has demonstrated that it is now relatively easy for anyone to fine-tune pre-trained AI models, even for potentially harmful purposes. With the low resource requirements and efficient techniques available, fine-tuning can be done with as few as 100 data points, and the cost of hosting and inference has become more manageable. This accessibility raises concerns about the potential misuse of AI models and the need for responsible development. The fine-tuning libraries and inference platforms have matured, making it easier for companies to host their own models and lessen their dependency on third-party services.
The potential risks and responsibilities of advanced AI technology
As AI models continue to advance, questions of AI alignment and safety become more critical. The exponential progress in AI technology and the proliferation of models like GPT-4 highlight the need for careful consideration of their implications. The power and capabilities of these models pose potential risks, and it becomes crucial to examine the values and sensibilities that guide their development. While AI has the potential for tremendous upside, it requires responsible decision-making and a conscious effort to ensure alignment with human values. The responsibility falls on the teams and individuals working on the frontier of AI to continuously question and navigate the trajectory of these technologies.
The challenges of transitioning to AGI and the importance of deliberate choices
With the advancements in AI, the question of AGI (artificial general intelligence) is becoming more tangible. While narrow AI models can be highly capable and useful in specific domains, the transition to AGI requires careful consideration and deliberate choices. The trajectory towards AGI needs to be approached with caution, allowing time for better understanding and decision-making. The responsibility lies with the teams at leading AI companies to ensure that the AGI being developed aligns with human values and is not rushed into existence without thorough examination and evaluation. The need for continuous questioning, discussion, and exploration of different forms of AGI becomes crucial in guiding its responsible development.
OpenAI says its mission is to build AGI — an AI system that is better than human beings at everything. Should the world trust them to do that safely?
That’s the central theme of today’s episode with Nathan Labenz — entrepreneur, AI scout, and host of The Cognitive Revolution podcast.
Links to learn more, video, highlights, and full transcript.
Nathan saw the AI revolution coming years ago, and, astonished by the research he was seeing, set aside his role as CEO of Waymark and made it his full-time job to understand AI capabilities across every domain. He has been obsessively tracking the AI world since — including joining OpenAI’s “red team” that probed GPT-4 to find ways it could be abused, long before it was public.
Whether OpenAI was taking AI safety seriously enough became a topic of dinner table conversation around the world after the shocking firing and reinstatement of Sam Altman as CEO last month.
Nathan’s view: it’s complicated. Discussion of this topic has often been heated, polarising, and personal. But Nathan wants to avoid that and simply lay out, in a way that is impartial and fair to everyone involved, what OpenAI has done right and how it could do better in his view.
When he started on the GPT-4 red team, the model would do anything from diagnose a skin condition to plan a terrorist attack without the slightest reservation or objection. When later shown a “Safety” version of GPT-4 that was almost the same, he approached a member of OpenAI’s board to share his concerns and tell them they really needed to try out GPT-4 for themselves and form an opinion.
In today’s episode, we share this story as Nathan told it on his own show, The Cognitive Revolution, which he did in the hope that it would provide useful background to understanding the OpenAI board’s reservations about Sam Altman, which to this day have not been laid out in any detail.
But while he feared throughout 2022 that OpenAI and Sam Altman didn’t understand the power and risk of their own system, he has since been repeatedly impressed, and came to think of OpenAI as among the better companies that could hypothetically be working to build AGI.
Their efforts to make GPT-4 safe turned out to be much larger and more successful than Nathan was seeing. Sam Altman and other leaders at OpenAI seem to sincerely believe they’re playing with fire, and take the threat posed by their work very seriously. With the benefit of hindsight, Nathan suspects OpenAI’s decision to release GPT-4 when it did was for the best.
On top of that, OpenAI has been among the most sane and sophisticated voices advocating for AI regulations that would target just the most powerful AI systems — the type they themselves are building — and that could make a real difference. They’ve also invested major resources into new ‘Superalignment’ and ‘Preparedness’ teams, while avoiding using competition with China as an excuse for recklessness.
At the same time, it’s very hard to know whether it’s all enough. The challenge of making an AGI safe and beneficial may require much more than they hope or have bargained for. Given that, Nathan poses the question of whether it makes sense to try to build a fully general AGI that can outclass humans in every domain at the first opportunity. Maybe in the short term, we should focus on harvesting the enormous possible economic and humanitarian benefits of narrow applied AI models, and wait until we not only have a way to build AGI, but a good way to build AGI — an AGI that we’re confident we want, which we can prove will remain safe as its capabilities get ever greater.
By threatening to follow Sam Altman to Microsoft before his reinstatement as OpenAI CEO, OpenAI’s research team has proven they have enormous influence over the direction of the company. If they put their minds to it, they’re also better placed than maybe anyone in the world to assess if the company’s strategy is on the right track and serving the interests of humanity as a whole. Nathan concludes that this power and insight only adds to the enormous weight of responsibility already resting on their shoulders.
In today’s extensive conversation, Nathan and host Rob Wiblin discuss not only all of the above, but also:
- Speculation about the OpenAI boardroom drama with Sam Altman, given Nathan’s interactions with the board when he raised concerns from his red teaming efforts.
- Which AI applications we should be urgently rolling out, with less worry about safety.
- Whether governance issues at OpenAI demonstrate AI research can only be slowed by governments.
- Whether AI capabilities are advancing faster than safety efforts and controls.
- The costs and benefits of releasing powerful models like GPT-4.
- Nathan’s view on the game theory of AI arms races and China.
- Whether it’s worth taking some risk with AI for huge potential upside.
- The need for more “AI scouts” to understand and communicate AI progress.
- And plenty more.
Producer and editor: Keiran Harris
Audio Engineering Lead: Ben Cordell
Technical editing: Milo McGuire and Dominic Armstrong
Transcriptions: Katy Moore
Remember Everything You Learn from Podcasts
