David Thorstad: Bounded Rationality and the Case Against Longtermism
The Gradient: Perspectives on AI
Delving into Bounded Rationality and the Singularity Hypothesis
This chapter explores the concept of bounded rationality and its impact on long-term thinking, emphasizing deliberation as essential for agents with limited cognitive abilities. It then shifts to discussing the singularity hypothesis, drawing parallels between ultra-intelligent machines and philosophical arguments for God's existence. The conversation touches on the challenges of defining general intelligence in AI and the need for sustained hyperbolic growth for the singularity to be feasible.
Episode 122
I spoke with Professor David Thorstad about:
* The practical difficulties of doing interdisciplinary work
* Why theories of human rationality should account for boundedness, heuristics, and other cognitive limitations
* why EA epistemics suck (ok, it’s a little more nuanced than that)
Professor Thorstad is an Assistant Professor of Philosophy at Vanderbilt University, a Senior Research Affiliate at the Global Priorities Institute at Oxford, and a Research Affiliate at the MINT Lab at Australian National University. One strand of his research asks how cognitively limited agents should decide what to do and believe. A second strand asks how altruists should use limited funds to do good effectively.
Reach me at editor@thegradient.pub for feedback, ideas, guest suggestions.
Subscribe to The Gradient Podcast: Apple Podcasts | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter
Outline:
* (00:00) Intro
* (01:15) David’s interest in rationality
* (02:45) David’s crisis of confidence, models abstracted from psychology
* (05:00) Blending formal models with studies of the mind
* (06:25) Interaction between academic communities
* (08:24) Recognition of and incentives for interdisciplinary work
* (09:40) Movement towards interdisciplinary work
* (12:10) The Standard Picture of rationality
* (14:11) Why the Standard Picture was attractive
* (16:30) Violations of and rebellion against the Standard Picture
* (19:32) Mistakes made by critics of the Standard Picture
* (22:35) Other competing programs vs Standard Picture
* (26:27) Characterizing Bounded Rationality
* (27:00) A worry: faculties criticizing themselves
* (29:28) Self-improving critique and longtermism
* (30:25) Central claims in bounded rationality and controversies
* (32:33) Heuristics and formal theorizing
* (35:02) Violations of Standard Picture, vindicatory epistemology
* (37:03) The Reason Responsive Consequentialist View (RRCV)
* (38:30) Objective and subjective pictures
* (41:35) Reason responsiveness
* (43:37) There are no epistemic norms for inquiry
* (44:00) Norms vs reasons
* (45:15) Arguments against epistemic nihilism for belief
* (47:30) Norms and self-delusion
* (49:55) Difficulty of holding beliefs for pragmatic reasons
* (50:50) The Gibbardian picture, inquiry as an action
* (52:15) Thinking how to act and thinking how to live — the power of inquiry
* (53:55) Overthinking and conducting inquiry
* (56:30) Is thinking how to inquire as an all-things-considered matter?
* (58:00) Arguments for the RRCV
* (1:00:40) Deciding on minimal criteria for the view, stereotyping
* (1:02:15) Eliminating stereotypes from the theory
* (1:04:20) Theory construction in epistemology and moral intuition
* (1:08:20) Refusing theories for moral reasons and disciplinary boundaries
* (1:10:30) The argument from minimal criteria, evaluating against competing views
* (1:13:45) Comparing to other theories
* (1:15:00) The explanatory argument
* (1:17:53) Parfit and Railton, norms of friendship vs utility
* (1:20:00) Should you call out your friend for being a womanizer
* (1:22:00) Vindicatory Epistemology
* (1:23:05) Panglossianism and meliorative epistemology
* (1:24:42) Heuristics and recognition-driven investigation
* (1:26:33) Rational inquiry leading to irrational beliefs — metacognitive processing
* (1:29:08) Stakes of inquiry and costs of metacognitive processing
* (1:30:00) When agents are incoherent, focuses on inquiry
* (1:32:05) Indirect normative assessment and its consequences
* (1:37:47) Against the Singularity Hypothesis
* (1:39:00) Superintelligence and the ontological argument
* (1:41:50) Hardware growth and general intelligence growth, AGI definitions
* (1:43:55) Difficulties in arguing for hyperbolic growth
* (1:46:07) Chalmers and the proportionality argument
* (1:47:53) Arguments for/against diminishing growth, research productivity, Moore’s Law
* (1:50:08) On progress studies
* (1:52:40) Improving research productivity and technology growth
* (1:54:00) Mistakes in the moral mathematics of existential risk, longtermist epistemics
* (1:55:30) Cumulative and per-unit risk
* (1:57:37) Back and forth with longtermists, time of perils
* (1:59:05) Background risk — risks we can and can’t intervene on, total existential risk
* (2:00:56) The case for longtermism is inflated
* (2:01:40) Epistemic humility and longtermism
* (2:03:15) Knowledge production — reliable sources, blog posts vs peer review
* (2:04:50) Compounding potential errors in knowledge
* (2:06:38) Group deliberation dynamics, academic consensus
* (2:08:30) The scope of longtermism
* (2:08:30) Money in effective altruism and processes of inquiry
* (2:10:15) Swamping longtermist options
* (2:12:00) Washing out arguments and justified belief
* (2:13:50) The difficulty of long-term forecasting and interventions
* (2:15:50) Theory of change in the bounded rationality program
* (2:18:45) Outro
Links:
* David’s homepage and Twitter and blog
* Papers mentioned/read
* Bounded rationality and inquiry
* Why bounded rationality (in epistemology)?
* Against the newer evidentialists
* The accuracy-coherence tradeoff in cognition
* There are no epistemic norms of inquiry
* Global priorities and effective altruism
* Against the singularity hypothesis (+ blog posts)
* Three mistakes in the moral mathematics of existential risk (+ blog posts)
Get full access to The Gradient at thegradientpub.substack.com/subscribe
Remember Everything You Learn from Podcasts
