Machine Learning Street Talk (MLST)

The Compendium - Connor Leahy and Gabriel Alfour

Mar 30, 2025

Connor Leahy and Gabriel Alfour, AI researchers from Conjecture, dive deep into the critical issues of Artificial Superintelligence (ASI) safety. They discuss the existential risks of uncontrolled AI advancements, warning that a superintelligent AI could dominate humanity as humans do less intelligent species. The conversation also touches on the need for robust institutional support and ethical governance to navigate the complexities of AI alignment with human values while critiquing prevailing ideologies like techno-feudalism.

01:37:10

Creator website

Episode guests

Gabriel Alfour

Connor Leahy

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Connor Leahy and Gabriel Alfour warn that uncontrolled AI development poses existential risks, necessitating a serious focus on safety governance.

The complexity of AI intelligence development, driven by algorithms rather than biological processes, complicates effective safety measures and predictions.

Establishing clear regulatory boundaries, such as banning untrustworthy ASI, is essential to mitigate the dangers of advanced AI technologies.

Public sentiment increasingly advocates for AI regulation, pressing policymakers to prioritize safety measures in response to mounting societal concerns.

Deep dives

The Urgency of AI Safety Research

The rapid development of artificial intelligence raises significant concerns about safety and the existential risks it poses to humanity. Current advancements are driven more by scaling up existing models rather than a deeper understanding of intelligence and its implications. There appears to be a race to create ever more powerful AI without sufficient consideration for the safety measures that should be in place. Establishing a focused research initiative, akin to a Manhattan Project for AI safety, is crucial for addressing these challenges and ensuring that AI development is safe and controlled.

Intro

2min

The Anticipated Dangers of AI Advancement

7min

The Complex Nature of Intelligence

23min

Techno-Feudalism and the Erosion of Humanist Values

2min

Order and Chaos: A Critique of Libertarian Perspectives

3min

The Intricacies of Ethical Humanism and Societal Flourishing

2min

Navigating the Labyrinth of AI Alignment

21min

Navigating Crisis Communication and AI Governance

29min

Regulating AI: Balancing Innovation and Safety

7min

Connor Leahy and Gabriel Alfour, AI researchers from Conjecture and authors of "The Compendium," joinus for a critical discussion centered on Artificial Superintelligence (ASI) safety and governance. Drawing from their comprehensive analysis in "The Compendium," they articulate a stark warning about the existential risks inherent in uncontrolled AI development, framing it through the lens of "intelligence domination"—where a sufficiently advanced AI could subordinate humanity, much like humans dominate less intelligent species.

SPONSOR MESSAGES:

***

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.

Goto https://tufalabs.ai/

***

TRANSCRIPT + REFS + NOTES:

https://www.dropbox.com/scl/fi/p86l75y4o2ii40df5t7no/Compendium.pdf?rlkey=tukczgf3flw133sr9rgss0pnj&dl=0

https://www.thecompendium.ai/

https://en.wikipedia.org/wiki/Connor_Leahy

https://www.conjecture.dev/about

https://substack.com/@gabecc

TOC:

1. AI Intelligence and Safety Fundamentals

[00:00:00] 1.1 Understanding Intelligence and AI Capabilities

[00:06:20] 1.2 Emergence of Intelligence and Regulatory Challenges

[00:10:18] 1.3 Human vs Animal Intelligence Debate

[00:18:00] 1.4 AI Regulation and Risk Assessment Approaches

[00:26:14] 1.5 Competing AI Development Ideologies

2. Economic and Social Impact

[00:29:10] 2.1 Labor Market Disruption and Post-Scarcity Scenarios

[00:32:40] 2.2 Institutional Frameworks and Tech Power Dynamics

[00:37:40] 2.3 Ethical Frameworks and AI Governance Debates

[00:40:52] 2.4 AI Alignment Evolution and Technical Challenges

3. Technical Governance Framework

[00:55:07] 3.1 Three Levels of AI Safety: Alignment, Corrigibility, and Boundedness

[00:55:30] 3.2 Challenges of AI System Corrigibility and Constitutional Models

[00:57:35] 3.3 Limitations of Current Boundedness Approaches

[00:59:11] 3.4 Abstract Governance Concepts and Policy Solutions

4. Democratic Implementation and Coordination

[00:59:20] 4.1 Governance Design and Measurement Challenges

[01:00:10] 4.2 Democratic Institutions and Experimental Governance

[01:14:10] 4.3 Political Engagement and AI Safety Advocacy

[01:25:30] 4.4 Practical AI Safety Measures and International Coordination

CORE REFS:

[00:01:45] The Compendium (2023), Leahy et al.

https://pdf.thecompendium.ai/the_compendium.pdf

[00:06:50] Geoffrey Hinton Leaves Google, BBC News

https://www.bbc.com/news/world-us-canada-65452940

[00:10:00] ARC-AGI, Chollet

https://arcprize.org/arc-agi

[00:13:25] A Brief History of Intelligence, Bennett

https://www.amazon.com/Brief-History-Intelligence-Humans-Breakthroughs/dp/0063286343

[00:25:35] Statement on AI Risk, Center for AI Safety

https://www.safe.ai/work/statement-on-ai-risk

[00:26:15] Machines of Love and Grace, Amodei

https://darioamodei.com/machines-of-loving-grace

[00:26:35] The Techno-Optimist Manifesto, Andreessen

https://a16z.com/the-techno-optimist-manifesto/

[00:31:55] Techno-Feudalism, Varoufakis

https://www.amazon.co.uk/Technofeudalism-Killed-Capitalism-Yanis-Varoufakis/dp/1847927270

[00:42:40] Introducing Superalignment, OpenAI

https://openai.com/index/introducing-superalignment/

[00:47:20] Three Laws of Robotics, Asimov

https://www.britannica.com/topic/Three-Laws-of-Robotics

[00:50:00] Symbolic AI (GOFAI), Haugeland

https://en.wikipedia.org/wiki/Symbolic_artificial_intelligence

[00:52:30] Intent Alignment, Christiano

https://www.alignmentforum.org/posts/HEZgGBZTpT4Bov7nH/mapping-the-conceptual-territory-in-ai-existential-safety

[00:55:10] Large Language Model Alignment: A Survey, Jiang et al.

http://arxiv.org/pdf/2309.15025

[00:55:40] Constitutional Checks and Balances, Bok

https://plato.stanford.edu/entries/montesquieu/

<trunc, see PDF>

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.

Machine Learning Street Talk (MLST)

The Compendium - Connor Leahy and Gabriel Alfour

Episode guests

Podcast summary created with Snipd AI

Quick takeaways

Deep dives

The Urgency of AI Safety Research

Understanding Intelligence: A Scientific Challenge

Caution in AI Regulations

The Role of Governance in AI Development

Identifying Critical Risks and Solutions

The Collective Action Problem

Public Perception and Pressure for Change

Remember Everything You Learn from Podcasts