The Gradient: Perspectives on AI
David Thorstad: Bounded Rationality and the Case Against Longtermism
Episode guests
Podcast summary created with Snipd AI
Quick takeaways
- Bounded rationality necessitates considering cognitive limitations in decision-making, challenging standard rationality models.
- Interdisciplinary research evolution involves recognizing, incentivizing, and evaluating collaborative efforts in academia.
- Rational inquiry entails responding to reasons and optimizing outcomes, emphasizing information sensitivity and consequentialist perspectives.
- Epistemic norms for inquiry are shifting towards a comprehensive approach, balancing rationality and decision-making complexities.
- Critiques of the Singularity Hypothesis focus on challenges of sustaining hyperbolic growth and the proportionality thesis, requiring robust arguments.
Deep dives
Synopsis of Bounded Rationality and Long-Termism
Bounded rationality, as discussed, emphasizes that people are not perfectly rational and outlines the Bounded Rationality program developed by David Thorstad, considering factors like accuracy, coherence, trade-offs, and epistemic norms in decision-making. The discussion delves into the intersection of bounded rationality and long-termism, highlighting the significance and controversies surrounding long-termism in academic discourse. Views on rationality and inquiry are shaped by a journey from formal perspectives in Bayesian statistics to crisis of confidence moments, leading to a search for models that incorporate psychology effectively.
Challenges to the Standard Rationality Models
The podcast episode delves into the challenges posed to standard rationality models, particularly in economics and decision theory, reflecting on the limitations of formal models in capturing human decision-making complexities. The narrative recounts experiences of conflicting perspectives between different academic groups, such as economists and psychologists, emphasizing the evolution towards interdisciplinary research approaches. The debate surrounding the recognition and evaluation of interdisciplinary work by academic institutions and journals is also scrutinized, shedding light on the complexities and merits of collaborative research efforts.
Exploring Reason-Responsive and Consequentialist Views
The conversation discusses reason-responsive and consequentialist views in rational inquiry, illustrating how rationality entails responding appropriately to reasons and engaging in actions that align with achieving optimal outcomes. The episode explores the concept of information sensitivity in determining norms for inquiry, emphasizing the consequentialist perspective on rational action. The dialogue emphasizes a shift towards evaluating inquiry through a broader lens that incorporates consequences and all-encompassing normative frameworks.
Evolving Perspectives on Epistemic Norms and Inquiry
A deep dive into the evolving perspectives on epistemic norms for inquiry unfolds, challenging conventional views and advocating for an alternative Gabardian picture where norms for inquiry are viewed as comprehensive norms governing action. The discussion navigates the intricacies of whether to overthink as a form of inquiry and addresses the meta-level inquiry conundrum, balancing rationality and decision-making strategies across various levels of inquiry. The narrative delves into the complexity of meta-cognitive considerations and the all-encompassing nature of inquiry in shaping outcomes and beliefs.
Argument for Diminishing Research Productivity and the Singularity Hypothesis
The Singularity Hypothesis posits hyperbolic growth in intelligence leading to an intelligence explosion. Critics like Chalmers raise concerns about sustaining this growth. Diminishing research productivity is a global phenomenon affecting sectors, including Moore's Law and agricultural research. Moore's Law experienced a 19 times drop in research productivity, indicating challenges in continuous growth. Concrete data on decreasing research efficiency presents a crucial argument against the sustainability of hyperbolic growth proposed in the Singularity Hypothesis.
Proportionality Thesis and Defining General Intelligence
The proportionality thesis, proposed by Chalmers, states that a percentage increase in a system's intelligence yields an equal increase in designing intelligence systems. This argument, often used in defense of Singularity, faces challenges in proving sustained hyperbolic growth. Definitions of 'general intelligence' vary, contributing to confusion in discussions. Chalmers' thesis focuses on exponential rather than hyperbolic growth, necessitating further support for asserting Singularity.
Argument for Hardware Growth and General Intelligence Growth
Distinguishing between hardware and general intelligence growth is vital when discussing the Singularity Hypothesis. Emphasizing hyperbolic growth as a requirement for Singularity, challenges arise in maintaining this trend. Defenders require solid arguments against diminishing growth rates to substantiate claims. The applicability of Chalmers' proportionality thesis in real-world scenarios, along with addressing diminishing research efficiency, is crucial to ensuring the viability of the Singularity Hypothesis.
Reflections on the Singularity Hypothesis and Diminishing Growth
Evaluating the Singularity Hypothesis involves addressing concerns around sustained hyperbolic growth and the proportionality thesis. Understanding diminishing research productivity, as seen in Moore's Law and agricultural research, provides insights into the challenges of continuous growth. Critically analyzing arguments for and against diminishing growth rates is essential in assessing the feasibility of the Singularity Hypothesis.
Diminishing Research Productivity Challenges in Various Domains
Research in fields like crop yield improvement and medical advancements is facing challenges in productivity due to the diminishing returns on research investments. As research progresses, it becomes increasingly difficult to achieve significant gains in areas crucial for human well-being and technological advancements. This trend reflects the struggle to maintain research advancements in critical domains, thus prompting a need for innovative approaches to boost research productivity.
Critique of Long-Termist Considerations and Existential Risk Assessments
Critics of long-termism emphasize the importance of engaging with the arguments presented by long-termists to evaluate their claims thoroughly. A critical examination of long-termist perspectives reveals potential oversights and mistakes in assessing existential risks. By highlighting areas such as focusing on cumulative versus per unit risk and overlooking background risks, critics aim to provoke discussions that lead to more comprehensive and accurate assessments of long-term considerations and risk mitigation strategies.
Episode 122
I spoke with Professor David Thorstad about:
* The practical difficulties of doing interdisciplinary work
* Why theories of human rationality should account for boundedness, heuristics, and other cognitive limitations
* why EA epistemics suck (ok, it’s a little more nuanced than that)
Professor Thorstad is an Assistant Professor of Philosophy at Vanderbilt University, a Senior Research Affiliate at the Global Priorities Institute at Oxford, and a Research Affiliate at the MINT Lab at Australian National University. One strand of his research asks how cognitively limited agents should decide what to do and believe. A second strand asks how altruists should use limited funds to do good effectively.
Reach me at editor@thegradient.pub for feedback, ideas, guest suggestions.
Subscribe to The Gradient Podcast: Apple Podcasts | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter
Outline:
* (00:00) Intro
* (01:15) David’s interest in rationality
* (02:45) David’s crisis of confidence, models abstracted from psychology
* (05:00) Blending formal models with studies of the mind
* (06:25) Interaction between academic communities
* (08:24) Recognition of and incentives for interdisciplinary work
* (09:40) Movement towards interdisciplinary work
* (12:10) The Standard Picture of rationality
* (14:11) Why the Standard Picture was attractive
* (16:30) Violations of and rebellion against the Standard Picture
* (19:32) Mistakes made by critics of the Standard Picture
* (22:35) Other competing programs vs Standard Picture
* (26:27) Characterizing Bounded Rationality
* (27:00) A worry: faculties criticizing themselves
* (29:28) Self-improving critique and longtermism
* (30:25) Central claims in bounded rationality and controversies
* (32:33) Heuristics and formal theorizing
* (35:02) Violations of Standard Picture, vindicatory epistemology
* (37:03) The Reason Responsive Consequentialist View (RRCV)
* (38:30) Objective and subjective pictures
* (41:35) Reason responsiveness
* (43:37) There are no epistemic norms for inquiry
* (44:00) Norms vs reasons
* (45:15) Arguments against epistemic nihilism for belief
* (47:30) Norms and self-delusion
* (49:55) Difficulty of holding beliefs for pragmatic reasons
* (50:50) The Gibbardian picture, inquiry as an action
* (52:15) Thinking how to act and thinking how to live — the power of inquiry
* (53:55) Overthinking and conducting inquiry
* (56:30) Is thinking how to inquire as an all-things-considered matter?
* (58:00) Arguments for the RRCV
* (1:00:40) Deciding on minimal criteria for the view, stereotyping
* (1:02:15) Eliminating stereotypes from the theory
* (1:04:20) Theory construction in epistemology and moral intuition
* (1:08:20) Refusing theories for moral reasons and disciplinary boundaries
* (1:10:30) The argument from minimal criteria, evaluating against competing views
* (1:13:45) Comparing to other theories
* (1:15:00) The explanatory argument
* (1:17:53) Parfit and Railton, norms of friendship vs utility
* (1:20:00) Should you call out your friend for being a womanizer
* (1:22:00) Vindicatory Epistemology
* (1:23:05) Panglossianism and meliorative epistemology
* (1:24:42) Heuristics and recognition-driven investigation
* (1:26:33) Rational inquiry leading to irrational beliefs — metacognitive processing
* (1:29:08) Stakes of inquiry and costs of metacognitive processing
* (1:30:00) When agents are incoherent, focuses on inquiry
* (1:32:05) Indirect normative assessment and its consequences
* (1:37:47) Against the Singularity Hypothesis
* (1:39:00) Superintelligence and the ontological argument
* (1:41:50) Hardware growth and general intelligence growth, AGI definitions
* (1:43:55) Difficulties in arguing for hyperbolic growth
* (1:46:07) Chalmers and the proportionality argument
* (1:47:53) Arguments for/against diminishing growth, research productivity, Moore’s Law
* (1:50:08) On progress studies
* (1:52:40) Improving research productivity and technology growth
* (1:54:00) Mistakes in the moral mathematics of existential risk, longtermist epistemics
* (1:55:30) Cumulative and per-unit risk
* (1:57:37) Back and forth with longtermists, time of perils
* (1:59:05) Background risk — risks we can and can’t intervene on, total existential risk
* (2:00:56) The case for longtermism is inflated
* (2:01:40) Epistemic humility and longtermism
* (2:03:15) Knowledge production — reliable sources, blog posts vs peer review
* (2:04:50) Compounding potential errors in knowledge
* (2:06:38) Group deliberation dynamics, academic consensus
* (2:08:30) The scope of longtermism
* (2:08:30) Money in effective altruism and processes of inquiry
* (2:10:15) Swamping longtermist options
* (2:12:00) Washing out arguments and justified belief
* (2:13:50) The difficulty of long-term forecasting and interventions
* (2:15:50) Theory of change in the bounded rationality program
* (2:18:45) Outro
Links:
* David’s homepage and Twitter and blog
* Papers mentioned/read
* Bounded rationality and inquiry
* Why bounded rationality (in epistemology)?
* Against the newer evidentialists
* The accuracy-coherence tradeoff in cognition
* There are no epistemic norms of inquiry
* Global priorities and effective altruism
* Against the singularity hypothesis (+ blog posts)
* Three mistakes in the moral mathematics of existential risk (+ blog posts)
Get full access to The Gradient at thegradientpub.substack.com/subscribe
Remember Everything You Learn from Podcasts
