AXRP - the AI X-risk Research Podcast

22 - Shard Theory with Quintin Pope

Jun 15, 2023

03:28:21

Creator website

AI Chapters

Episode notes

Introduction

2min

The Importance of Understanding Human Value Formation

4min

The Benefits of Replicating Causal Processes in Deep Learning

3min

The Convergence of Deep Learning and the Human Brain

3min

How AI Learning Will Look Like Human Learning Within the Lifetime

5min

The Difference Between Values and AI Alignment

3min

The Advantages of Aligning AI to Ambiguity

3min

The Different Intuitions Behind Deep Learning

2min

The Orthogonality of Deep Learning

2min

10.

The Importance of Introspection in Human Value Formation

3min

11.

The Multiple Cortices of the Brain

2min

12.

The Complexity of Self-Supervised Learning

2min

13.

The Role of Reward in Learning

2min

14.

The Importance of Reward in Human Learning

3min

15.

The Role of Genome in Reinforcement Learning

3min

16.

The Role of Conditional Circuitry in Reward Learning

2min

17.

The Role of Reward Signals in Language Modeling

4min

18.

How the Genome Influences Human Learning

4min

19.

The Genetic Influence of Religion

4min

20.

The Role of EEG in Learning Emotions

4min

21.

Genome-Specified Reward Circuitry for Emotions

2min

22.

The Brain and the Human World Model

2min

23.

The Similarity of the Human Brain to Deep Learning

4min

24.

The Effects of Noise on the Learning Process

3min

25.

The Importance of Noise in Deep Learning

2min

26.

The Inductive Biases of Machine Learning Systems

1min

27.

The Parameter Function Map in Deep Learning Systems

6min

28.

The Inductive Biases of Neural Networks

3min

29.

The Differences Between Blind and Sighted Humans

3min

30.

The Shard Theory of Human Values

3min

31.

The Role of Values in Decision Making

2min

32.

The Role of Shards in Decision Making

2min

33.

The Role of Shards in Decision Making

3min

34.

The Effects of Shard Theory on Food Preferences

2min

35.

The Importance of Symmetries in Food Choices

2min

36.

The Limits of Shard Theory in Deep Learning

4min

37.

How to Use Learned Loss Functions to Optimize Internal Cognitive States

2min

38.

The Problem With Expected Utility Functions in Deep Learning

5min

39.

The Importance of Utility Optimization in Brain Behavior

2min

40.

The Constraints of Hard Theory on Utility Optimization

3min

41.

The Convergence of Short Theory Perspectives on Human Value Formation

5min

42.

The Behavior of Deep Learning Models

6min

43.

The Implications of Char-D Theory for Deep Learning

6min

44.

The Open Eye Model and Deep Learning

6min

45.

The Role of Hard Theory in Shapeing Our Anticipations About Stories

2min

46.

The Problem With Rewarding Ais for Tricking

2min

47.

The Higher Frequency Behavior of AI

3min

48.

The Tension Between Things That Are Highly Rated by Humans and Things That Actually Are Good

2min

49.

The Roar Function and the Behavior of the Machine Learning Agent

6min

50.

The Discriminator Generator Gap in AI Capabilities

3min

51.

The Benefits of Alignment Over Capabilities

3min

52.

The Importance of Data in Science

2min

53.

The Connection Between the Generator and Discriminator Gap in AI Knowledge

2min

54.

The Implications of Chart Theory for AI

4min

55.

The Power of Self-Supervised Learning

2min

56.

The Singular Learning Theory and the Learning Dynamics

2min

57.

The Differences Between Self-Supervised Learning and RL in Deep Learning

2min

58.

The Importance of Singular Learning Theory in Deep Learning

3min

59.

The Role of Chart Theory in Human Value Formation

4min

60.

The Future of Cognitive Flexibility in AI

3min

61.

The Research Community on Chart Theory

6min

62.

The Effect of Training Processes on Downstream Behaviors

3min

63.

The Importance of Language Model Alignment

6min

64.

The Different Applications of Rl in Language Models

3min

65.

The Importance of Consequences in Deep Learning

5min

66.

The Prosaic Alignment of Deep Learning

4min

67.

How to Follow Your Research on Hard Theory

2min

What can we learn about advanced deep learning systems by understanding how humans learn and form values over their lifetimes? Will superhuman AI look like ruthless coherent utility optimization, or more like a mishmash of contextually activated desires? This episode's guest, Quintin Pope, has been thinking about these questions as a leading researcher in the shard theory community. We talk about what shard theory is, what it says about humans and neural networks, and what the implications are for making AI safe.

Patreon: patreon.com/axrpodcast

Ko-fi: ko-fi.com/axrpodcast

Episode art by Hamish Doodles: hamishdoodles.com

Topics we discuss, and timestamps:

- 0:00:42 - Why understand human value formation?

- 0:19:59 - Why not design methods to align to arbitrary values?

- 0:27:22 - Postulates about human brains

- 0:36:20 - Sufficiency of the postulates

- 0:44:55 - Reinforcement learning as conditional sampling

- 0:48:05 - Compatibility with genetically-influenced behaviour

- 1:03:06 - Why deep learning is basically what the brain does

- 1:25:17 - Shard theory

- 1:38:49 - Shard theory vs expected utility optimizers

- 1:54:45 - What shard theory says about human values

- 2:05:47 - Does shard theory mean we're doomed?

- 2:18:54 - Will nice behaviour generalize?

- 2:33:48 - Does alignment generalize farther than capabilities?

- 2:42:03 - Are we at the end of machine learning history?

- 2:53:09 - Shard theory predictions

- 2:59:47 - The shard theory research community

- 3:13:45 - Why do shard theorists not work on replicating human childhoods?

- 3:25:53 - Following shardy research

The transcript: axrp.net/episode/2023/06/15/episode-22-shard-theory-quintin-pope.html

Shard theorist links:

- Quintin's LessWrong profile: lesswrong.com/users/quintin-pope

- Alex Turner's LessWrong profile: lesswrong.com/users/turntrout

- Shard theory Discord: discord.gg/AqYkK7wqAG

- EleutherAI Discord: discord.gg/eleutherai

Research we discuss:

- The Shard Theory Sequence: lesswrong.com/s/nyEFg3AuJpdAozmoX

- Pretraining Language Models with Human Preferences: arxiv.org/abs/2302.08582

- Inner alignment in salt-starved rats: lesswrong.com/posts/wcNEXDHowiWkRxDNv/inner-alignment-in-salt-starved-rats

- Intro to Brain-like AGI Safety Sequence: lesswrong.com/s/HzcM2dkCq7fwXBej8

- Brains and transformers:

- The neural architecture of language: Integrative modeling converges on predictive processing: pnas.org/doi/10.1073/pnas.2105646118

- Brains and algorithms partially converge in natural language processing: nature.com/articles/s42003-022-03036-1

- Evidence of a predictive coding hierarchy in the human brain listening to speech: nature.com/articles/s41562-022-01516-2

- Singular learning theory explainer: Neural networks generalize because of this one weird trick: lesswrong.com/posts/fovfuFdpuEwQzJu2w/neural-networks-generalize-because-of-this-one-weird-trick

- Singular learning theory links: metauni.org/slt/

- Implicit Regularization via Neural Feature Alignment, aka circles in the parameter-function map: arxiv.org/abs/2008.00938

- The shard theory of human values: lesswrong.com/s/nyEFg3AuJpdAozmoX/p/iCfdcxiyr2Kj8m8mT

- Predicting inductive biases of pre-trained networks: openreview.net/forum?id=mNtmhaDkAr

- Understanding and controlling a maze-solving policy network, aka the cheese vector: lesswrong.com/posts/cAC4AXiNC5ig6jQnc/understanding-and-controlling-a-maze-solving-policy-network

- Quintin's Research agenda: Supervising AIs improving AIs: lesswrong.com/posts/7e5tyFnpzGCdfT4mR/research-agenda-supervising-ais-improving-ais

- Steering GPT-2-XL by adding an activation vector: lesswrong.com/posts/5spBue2z2tw4JuDCx/steering-gpt-2-xl-by-adding-an-activation-vector

Links for the addendum on mesa-optimization skepticism:

- Quintin's response to Yudkowsky arguing against AIs being steerable by gradient descent: lesswrong.com/posts/wAczufCpMdaamF9fy/my-objections-to-we-re-all-gonna-die-with-eliezer-yudkowsky#Yudkowsky_argues_against_AIs_being_steerable_by_gradient_descent_

- Quintin on why evolution is not like AI training: lesswrong.com/posts/wAczufCpMdaamF9fy/my-objections-to-we-re-all-gonna-die-with-eliezer-yudkowsky#Edit__Why_evolution_is_not_like_AI_training

- Evolution provides no evidence for the sharp left turn: lesswrong.com/posts/hvz9qjWyv8cLX9JJR/evolution-provides-no-evidence-for-the-sharp-left-turn

- Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasets: arxiv.org/abs/1905.10854

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.