"The Cognitive Revolution" | AI Builders, Researchers, and Live Player Analysis

Beyond Preference Alignment: Teaching AIs to Play Roles & Respect Norms, with Tan Zhi Xuan

Nov 30, 2024

In this discussion, Tan Zhi Xuan, an MIT PhD student specializing in AI alignment, critiques traditional preference-based methods. They propose role-based AI systems shaped by social consensus, emphasizing the necessity of aligning AI with societal norms instead of mere preferences. The conversation touches on how AI can learn ethical standards through Bayesian reasoning and the exploration of self-other overlap to enhance cooperation. Xuan's innovative insights pave the way for a safer, more socially aware approach to AI development.

01:57:12

Creator website

Episode guests

Tan Zhi Xuan

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

The podcast critiques preference-based AI alignment, advocating for role-based systems that adhere to social norms and consensus.

Incorporating diverse ethical frameworks, including Eastern and Western philosophies, is crucial for developing responsible AI behaviors in different contexts.

AI systems can learn social norms through Bayesian rule induction in Markov games, enhancing cooperation and addressing collective resource management issues.

A gradual integration of AI into society, emphasizing decentralization and continuous learning, is necessary to ensure safety and ethical alignment in development.

Deep dives

Critique of the Preferences Paradigm

The episode critiques the prevalent view that AI systems should align with human preferences, highlighting the limitations of expected utility maximization. It argues that learned utility functions derived from preference data often fail to accurately represent what individuals truly desire, leading to potential over-optimization issues. Instead, it advocates for a focus on defining clear social roles and normative standards for AI behavior, similar to professionals adhering to societal expectations. This shift aims to ensure AI systems meet minimal moral standards rather than strictly optimizing for vague and inconsistent preferences.

Intro

5min

Navigating AI Alignment and State Dynamics

12min

Understanding AI Alignment: The Preferentist Approach

4min

Aligning AI with Human Norms

13min

Reconceptualizing AI Roles and Norms for Effective Alignment

6min

Navigating AI Roles and Responsibilities

21min

Aligning AI with Human Values

14min

Aligning AI with Social Norms

23min

Navigating AI Alignment and Cooperation

17min

In this episode of The Cognitive Revolution, Nathan explores groundbreaking perspectives on AI alignment with MIT PhD student Tan Zhi Xuan. We dive deep into Xuan's critique of preference-based AI alignment and their innovative proposal for role-based AI systems guided by social consensus. The conversation extends into their fascinating work on how AI agents can learn social norms through Bayesian rule induction. Join us for an intellectually stimulating discussion that bridges philosophical theory with practical implementation in AI development.

Check out:

"Beyond Preferences in AI Alignment" paper: https://arxiv.org/pdf/2408.16984

"Learning and Sustaining Shared Normative Systems via Bayesian Rule Induction in Markov Games" paper: https://arxiv.org/pdf/2402.13399

Help shape our show by taking our quick listener survey at https://bit.ly/TurpentinePulse

SPONSORS:

Notion: Notion offers powerful workflow and automation templates, perfect for streamlining processes and laying the groundwork for AI-driven automation. With Notion AI, you can search across thousands of documents from various platforms, generating highly relevant analysis and content tailored just for you - try it for free at https://notion.com/cognitiverevolution

Weights & Biases RAG++: Advanced training for building production-ready RAG applications. Learn from experts to overcome LLM challenges, evaluate systematically, and integrate advanced features. Includes free Cohere credits. Visit https://wandb.me/cr to start the RAG++ course today.

Oracle Cloud Infrastructure (OCI): Oracle's next-generation cloud platform delivers blazing-fast AI and ML performance with 50% less for compute and 80% less for outbound networking compared to other cloud providers13. OCI powers industry leaders with secure infrastructure and application development capabilities. New U.S. customers can get their cloud bill cut in half by switching to OCI before December 31, 2024 at https://oracle.com/cognitive

RECOMMENDED PODCAST:

Unpack Pricing - Dive into the dark arts of SaaS pricing with Metronome CEO Scott Woody and tech leaders. Learn how strategic pricing drives explosive revenue growth in today's biggest companies like Snowflake, Cockroach Labs, Dropbox and more.

Apple: https://podcasts.apple.com/us/podcast/id1765716600

Spotify: https://open.spotify.com/show/38DK3W1Fq1xxQalhDSueFg

CHAPTERS:

(00:00:00) Teaser

(00:01:09) About the Episode

(00:04:25) Guest Intro

(00:06:25) Xuan's Background

(00:12:03) AI Near-Term Outlook

(00:17:32) Sponsors: Notion | Weights & Biases RAG++

(00:20:18) Alignment Approaches

(00:26:11) Critiques of RLHF

(00:34:40) Sponsors: Oracle Cloud Infrastructure (OCI)

(00:35:50) Beyond Preferences

(00:40:27) Roles and AI Systems

(00:45:19) What AI Owes Us

(00:51:52) Drexler's AI Services

(01:01:08) Constitutional AI

(01:09:43) Technical Approach

(01:22:01) Norms and Deviations

(01:32:31) Norm Decay

(01:38:06) Self-Other Overlap

(01:44:05) Closing Thoughts

(01:54:23) Outro

SOCIAL LINKS:

Website: https://www.cognitiverevolution.ai

Twitter (Podcast): https://x.com/cogrev_podcast

Twitter (Nathan): https://x.com/labenz

LinkedIn: https://www.linkedin.com/in/nathanlabenz/

Youtube: https://www.youtube.com/@CognitiveRevolutionPodcast

Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431

Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more