AXRP - the AI X-risk Research Podcast

38.8 - David Duvenaud on Sabotage Evaluations and the Post-AGI Future

Mar 1, 2025

In this discussion, David Duvenaud, a University of Toronto professor specializing in probabilistic deep learning and AI safety at Anthropic, dives into the challenges of assessing whether AI models could sabotage human decisions. He shares insights on the complexities of sabotage evaluations and strategies needed for effective oversight. The conversation shifts to the societal impacts of a post-AGI world, reflecting on potential job implications and the delicate balance between AI advancement and prioritizing human values.

20:42

Creator website

Episode guests

David Duvenaud

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

David Duvenaud discusses the complexities of sabotage evaluations, emphasizing the need for holistic approaches to assess AI models' potential risks.

He raises concerns about humanity's diminishing influence in a post-AGI world, advocating for societal adaptations to keep humans central in decision-making processes.

Deep dives

David Duveneau's Contributions to AI Safety

David Duveneau, a professor at the University of Toronto, has shifted his focus towards AI safety since 2017. His recent work includes a year and a half sabbatical at Anthropic, where he led a team known as Alignment Evals. This team developed evaluations to assess whether AI models could secretly sabotage human decision-making or mislead evaluators about their capabilities. The aim was to establish a framework for understanding and mitigating potential risks associated with advanced AI models.

Intro

2min

Evaluating AI Sabotage Risks

11min

Reflections on Post-AGI Societal Implications

2min

The Limits of AI Advocacy in a Competitive Future

2min

Navigating the Trade-Offs of AI and Human Values

3min

In this episode, I chat with David Duvenaud about two topics he's been thinking about: firstly, a paper he wrote about evaluating whether or not frontier models can sabotage human decision-making or monitoring of the same models; and secondly, the difficult situation humans find themselves in in a post-AGI future, even if AI is aligned with human intentions.

Patreon: https://www.patreon.com/axrpodcast

Ko-fi: https://ko-fi.com/axrpodcast

Transcript: https://axrp.net/episode/2025/03/01/episode-38_8-david-duvenaud-sabotage-evaluations-post-agi-future.html

FAR.AI: https://far.ai/

FAR.AI on X (aka Twitter): https://x.com/farairesearch

FAR.AI on YouTube: @FARAIResearch

The Alignment Workshop: https://www.alignment-workshop.com/

Topics we discuss, and timestamps:

01:42 - The difficulty of sabotage evaluations

05:23 - Types of sabotage evaluation

08:45 - The state of sabotage evaluations

12:26 - What happens after AGI?