The Nonlinear Library LW - The Intentional Stance, LLMs Edition by Eleni Angelou
May 1, 2024
15:51
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Intentional Stance, LLMs Edition, published by Eleni Angelou on May 1, 2024 on LessWrong.
In memoriam of Daniel C. Dennett.
tl;dr: I sketch out what it means to apply Dennett's Intentional Stance to LLMs. I argue that the intentional vocabulary is already ubiquitous in experimentation with these systems therefore what is missing is the theoretical framework to justify this usage. I aim to make up for that and explain why the intentional stance is the best available explanatory tool for LLM behavior.
Choosing Between Stances
Why choose the intentional stance?
It seems natural to employ or ascribe cognitive states to AI models starting from the field's terminology, most prominently by calling it "machine learning" (Hagendorff 2023). This is very much unlike how other computer programs are treated.
When programmers write software, they typically understand it in terms of what they designed it to execute (design stance) or simply make sense of it considering its physical properties, such as the materials it was made of or the various electrical signals processing in its circuitry (physical stance). As I note, it is not that we cannot use Dennett's other two stances (Dennett 1989) to talk about these systems.
It is rather that neither of them constitutes the best explanatory framework for interacting with LLMs.
To illustrate this, consider the reverse example. It is possible to apply the intentional stance to a hammer although this does not generate any new information or optimally explain the behavior of the tool. What seems to be apt for making sense of how hammers operate instead is the design stance. This is just as applicable to other computer programs-tools. To use a typical program, there is no need to posit intentional states.
Unlike LLMs, users do not engage in human-like conversation with the software.
More precisely, the reason why neither the design nor the physical stance is sufficient to explain and predict the behavior of LLMs is because state-of-the-art LLM outputs are in practice indistinguishable from those of human agents (Y. Zhou et al. 2022). It is possible to think about LLMs as trained systems or as consisting of graphic cards and neural network layers, but these hardly make any difference when one attempts to prompt them and make them helpful for conversation and problem-solving.
What is more, machine learning systems like LLMs are not programmed to execute a task but are rather trained to find the policy that will execute the task. In other words, developers are not directly coding the information required to solve the problem they are using the AI for: they train the system to find the solution on its own. This requires for the model to possess all the necessary concepts.
In that sense, dealing with LLMs is more akin to studying a biological organism that is under development or perhaps raising a child, and less like building a tool the use of which is well-understood prior to the system's interaction with its environment. The LLM can learn from feedback and "change its mind" about the optimal policy to go about its task which is not the case for the standard piece of software. Moreover, LLMs seem to possess concepts.
Consequently, there is a distinction to be drawn between tool-like and agent-like programs. Judging on a behavioral basis, LLMs fall into the second category. This conclusion renders the intentional stance (Dennett 1989) practically indispensable for the evaluation of LLMs on a behavioral basis.
Folk Psychology for LLMs
What kind of folk psychology should we apply to LLMs? Do they have beliefs, desires, and goals?
LLMs acquire "beliefs" from their training distribution, since they do not memorize or copy any text from it when outputting their results - at least no more than human writers and speakers do. They must, as a result, ...
In memoriam of Daniel C. Dennett.
tl;dr: I sketch out what it means to apply Dennett's Intentional Stance to LLMs. I argue that the intentional vocabulary is already ubiquitous in experimentation with these systems therefore what is missing is the theoretical framework to justify this usage. I aim to make up for that and explain why the intentional stance is the best available explanatory tool for LLM behavior.
Choosing Between Stances
Why choose the intentional stance?
It seems natural to employ or ascribe cognitive states to AI models starting from the field's terminology, most prominently by calling it "machine learning" (Hagendorff 2023). This is very much unlike how other computer programs are treated.
When programmers write software, they typically understand it in terms of what they designed it to execute (design stance) or simply make sense of it considering its physical properties, such as the materials it was made of or the various electrical signals processing in its circuitry (physical stance). As I note, it is not that we cannot use Dennett's other two stances (Dennett 1989) to talk about these systems.
It is rather that neither of them constitutes the best explanatory framework for interacting with LLMs.
To illustrate this, consider the reverse example. It is possible to apply the intentional stance to a hammer although this does not generate any new information or optimally explain the behavior of the tool. What seems to be apt for making sense of how hammers operate instead is the design stance. This is just as applicable to other computer programs-tools. To use a typical program, there is no need to posit intentional states.
Unlike LLMs, users do not engage in human-like conversation with the software.
More precisely, the reason why neither the design nor the physical stance is sufficient to explain and predict the behavior of LLMs is because state-of-the-art LLM outputs are in practice indistinguishable from those of human agents (Y. Zhou et al. 2022). It is possible to think about LLMs as trained systems or as consisting of graphic cards and neural network layers, but these hardly make any difference when one attempts to prompt them and make them helpful for conversation and problem-solving.
What is more, machine learning systems like LLMs are not programmed to execute a task but are rather trained to find the policy that will execute the task. In other words, developers are not directly coding the information required to solve the problem they are using the AI for: they train the system to find the solution on its own. This requires for the model to possess all the necessary concepts.
In that sense, dealing with LLMs is more akin to studying a biological organism that is under development or perhaps raising a child, and less like building a tool the use of which is well-understood prior to the system's interaction with its environment. The LLM can learn from feedback and "change its mind" about the optimal policy to go about its task which is not the case for the standard piece of software. Moreover, LLMs seem to possess concepts.
Consequently, there is a distinction to be drawn between tool-like and agent-like programs. Judging on a behavioral basis, LLMs fall into the second category. This conclusion renders the intentional stance (Dennett 1989) practically indispensable for the evaluation of LLMs on a behavioral basis.
Folk Psychology for LLMs
What kind of folk psychology should we apply to LLMs? Do they have beliefs, desires, and goals?
LLMs acquire "beliefs" from their training distribution, since they do not memorize or copy any text from it when outputting their results - at least no more than human writers and speakers do. They must, as a result, ...