Prof. Subbarao Kambhampati - LLMs don't reason, they memorize (ICML2024 2/13)
Jul 29, 2024
auto_awesome
In this engaging discussion, Subbarao Kambhampati, a Professor at Arizona State University specializing in AI, tackles the limitations of large language models. He argues that these models primarily memorize rather than reason, raising questions about their reliability. Kambhampati explores the need for hybrid approaches that combine LLMs with external verification systems to ensure accuracy. He also delves into the distinctions between human reasoning and LLM capabilities, emphasizing the importance of critical skepticism in AI research.
Prof. Kambhampati highlights that LLMs excel in language generation but fundamentally lack genuine reasoning capabilities and logical verification.
The limitations of LLMs are evident in their reliance on memorization instead of independent reasoning, particularly in complex tasks like planning.
A hybrid approach integrating LLMs with external verification mechanisms can enhance their performance in logical reasoning and factual correctness.
Deep dives
The Nature of Reasoning and Answering
When faced with a reasoning question, it can be difficult to distinguish between answering from memory and genuine reasoning. An example is the question of why manhole covers are round, where the initial answer required understanding that other shapes could fall through the hole, illustrating first principles reasoning. Now, many people might respond based on research or prior knowledge, which only reflects preparation rather than reasoning ability. This highlights the uniqueness of human reasoning compared to large language models (LLMs), which excel in language generation but lack true reasoning capabilities.
Limitations of Large Language Models
LLMs are compared to n-gram models, functioning primarily as advanced retrieval systems rather than genuine reasoning entities. These models utilize statistical patterns to predict the next word, but they cannot verify or reason through complex problems autonomously. As such, LLMs cannot derive new knowledge but are proficient at completing language prompts based on training data. This discrepancy emphasizes a significant gap in their capabilities, particularly in tasks requiring logical deduction or first principles reasoning.
The Importance of Distinguishing Reasoning from Retrieval
A challenge arises in interpreting LLM performance on reasoning tasks since their successful responses may stem from retrieval rather than reasoning. For example, LLMs often perform well on standardized tests because these tests draw from a limited question bank, providing candidates opportunities to have previously encountered the questions. In the context of planning tasks, such as block stacking, LLMs exhibit poor performance when faced with variations in predicate names, revealing their reliance on memorized patterns rather than independent reasoning. This reinforces the idea that increments in model size do not necessarily lead to enhanced reasoning abilities.
Creative Generation Versus Analytical Verification
While LLMs demonstrate strength in creative idea generation, they falter in tasks requiring logical verification and deductive reasoning. A compelling argument suggests that LLMs are excellent at producing stylistically engaging content but lack accuracy in factual correctness. Creative challenges benefit from LLM output, but ensuring that results meet factual correctness demands verification through external systems or checks. The lack of internal verification within LLMs necessitates a modular approach where their generative capabilities are supported by other systems providing correctness verification.
LLM Modulo Architecture as a Solution
The LLM Modulo architecture integrates LLMs with verification mechanisms to enhance planning and reasoning capacities. By incorporating external critics and verifiers, this framework enhances the generative capabilities of LLMs and provides a means to achieve guarantees on correctness. In practice, this system employs LLMs to generate potential plans, which are then evaluated against established criteria by external verifiers, ensuring the output meets required standards. This approach exemplifies how combining LLMs with traditional verification can lead to improved performance in complex reasoning tasks.
Future Directions and the Role of Skepticism
As AI research evolves, there’s an increasing need for skepticism among researchers, particularly regarding claims about LLM capabilities. A crucial takeaway is the importance of verifying findings, recognizing that a positive result does not constitute proof of overall effectiveness. Emphasizing negative results can provide insights into limitations, fostering a more rigorous understanding of what LLMs can achieve. This balanced approach should guide future research, ensuring claims are substantiated and accurately reflect the strengths and weaknesses of existing models.
Prof. Subbarao Kambhampati argues that while LLMs are impressive and useful tools, especially for creative tasks, they have fundamental limitations in logical reasoning and cannot provide guarantees about the correctness of their outputs. He advocates for hybrid approaches that combine LLMs with external verification systems.
MLST is sponsored by Brave:
The Brave Search API covers over 20 billion webpages, built from scratch without Big Tech biases or the recent extortionate price hikes on search API access. Perfect for AI model training and retrieval augmentated generation. Try it now - get 2,000 free queries monthly at http://brave.com/api.
TOC (sorry the ones baked into the MP3 were wrong apropos due to LLM hallucination!)
