Yudkowsky, failure modes, and paths forward

Emmett agrees catastrophic tool failure is real but defends organic alignment as a possibly viable alternative.

Play episode from 01:10:15

chevron_right

Transcript

chevron_right

Transcript

Episode notes

Emmett Shear and Séb Krier debate whether today’s AI alignment paradigm—focused on control and instruction-following—is fundamentally flawed. PSA for AI builders: Interested in alignment, governance, or AI safety? Learn more about the MATS Summer 2026 Fellowship and submit your name to be notified when applications open: https://matsprogram.org/s26-tcr. They explore what changes if advanced AIs are better understood as beings with their own values, and why current control methods could drift toward something like slavery. The conversation dives into “organic alignment,” multi-agent simulations, evolving cooperation, and the possibility of AI moral standing as systems gain memory and continual learning.

Sponsors:

MATS:

MATS is a fully funded 12-week research program pairing rising talent with top mentors in AI alignment, interpretability, security, and governance. Apply for the next cohort at https://matsprogram.org/s26-tcr

Tasklet:

Tasklet is an AI agent that automates your work 24/7; just describe what you want in plain English and it gets the job done. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai

Agents of Scale:

Agents of Scale is a podcast from Zapier CEO Wade Foster, featuring conversations with C-suite leaders who are leading AI transformation. Subscribe to the show wherever you get your podcasts

Shopify:

Shopify powers millions of businesses worldwide, handling 10% of U.S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive

CHAPTERS:

(00:00) About the Episode

(03:44) Defining organic AI alignment

(14:48) Technical vs value alignment (Part 1)

(19:55) Sponsors: MATS | Tasklet

(22:56) Technical vs value alignment (Part 2) (Part 1)

(31:34) Sponsors: Agents of Scale | Shopify

(34:21) Technical vs value alignment (Part 2) (Part 2)

(34:22) Labs, tools, and beings

(43:22) AI personhood and consciousness

(56:53) Safe futures and Softmax

(01:04:17) Chatbots, mirrors, simulations