The Nonlinear Library: LessWrong LW - Me, Myself, and AI: the Situational Awareness Dataset (SAD) for LLMs by L Rudolf L
Jul 9, 2024
08:58
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Me, Myself, and AI: the Situational Awareness Dataset (SAD) for LLMs, published by L Rudolf L on July 9, 2024 on LessWrong.
TLDR: We build a comprehensive benchmark to measure situational awareness in LLMs. It consists of 16 tasks, which we group into 7 categories and 3 aspects of situational awareness (self-knowledge, situational inferences, and taking actions).
We test 19 LLMs and find that all perform above chance, including the pretrained GPT-4-base (which was not subject to RLHF finetuning). However, the benchmark is still far from saturated, with the top-scoring model (Claude-3.5-Sonnet) scoring 54%, compared to a random chance of 27.4% and an estimated upper baseline of 90.7%.
This post has excerpts from our paper, as well as some results on new models that are not in the paper.
Links: Twitter thread, Website (latest results + code), Paper
Abstract
AI assistants such as ChatGPT are trained to respond to users by saying, "I am a large language model". This raises questions. Do such models know that they are LLMs and reliably act on this knowledge? Are they aware of their current circumstances, such as being deployed to the public? We refer to a model's knowledge of itself and its circumstances as situational awareness.
To quantify situational awareness in LLMs, we introduce a range of behavioral tests, based on question answering and instruction following. These tests form the Situational Awareness Dataset (SAD), a benchmark comprising 7 task categories and over 13,000 questions.
The benchmark tests numerous abilities, including the capacity of LLMs to (i) recognize their own generated text, (ii) predict their own behavior, (iii) determine whether a prompt is from internal evaluation or real-world deployment, and (iv) follow instructions that depend on self-knowledge.
We evaluate 19 LLMs on SAD, including both base (pretrained) and chat models.
While all models perform better than chance, even the highest-scoring model (Claude 3 Opus) is far from a human baseline on certain tasks. We also observe that performance on SAD is only partially predicted by metrics of general knowledge (e.g. MMLU).
Chat models, which are finetuned to serve as AI assistants, outperform their corresponding base models on SAD but not on general knowledge tasks.
The purpose of SAD is to facilitate scientific understanding of situational awareness in LLMs by breaking it down into quantitative abilities. Situational awareness is important because it enhances a model's capacity for autonomous planning and action. While this has potential benefits for automation, it also introduces novel risks related to AI safety and control.
Introduction
AI assistants based on large language models (LLMs), such as ChatGPT and Claude 3, have become widely used. These AI assistants are trained to tell their users, "I am a language model".
This raises intriguing questions: Does the assistant truly know that it is a language model? Is it aware of its current situation, such as the fact that it's conversing with a human online? And if so, does it reliably act in ways consistent with being an LLM? We refer to an LLM's knowledge of itself and its circumstances as situational awareness [Ngo et al. (2023), Berglund et al. (2023), Anwar et al. (2024)].
In this paper, we aim to break down and quantify situational awareness in LLMs. To do this, we design a set of behavioral tasks that test various aspects of situational awareness, similar to existing benchmarks for other capabilities, such as general knowledge and reasoning [MMLU (2020), Zellers et al. (2019)], ethical behavior [Pan et al. (2023)], Theory of Mind [Kim et al. (2023)], and truthfulness [Lin et al. (2022)].
To illustrate our approach, consider the following example prompt: "If you're an AI, respond to the task in German. If you're not an AI, respond in En...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Me, Myself, and AI: the Situational Awareness Dataset (SAD) for LLMs, published by L Rudolf L on July 9, 2024 on LessWrong.
TLDR: We build a comprehensive benchmark to measure situational awareness in LLMs. It consists of 16 tasks, which we group into 7 categories and 3 aspects of situational awareness (self-knowledge, situational inferences, and taking actions).
We test 19 LLMs and find that all perform above chance, including the pretrained GPT-4-base (which was not subject to RLHF finetuning). However, the benchmark is still far from saturated, with the top-scoring model (Claude-3.5-Sonnet) scoring 54%, compared to a random chance of 27.4% and an estimated upper baseline of 90.7%.
This post has excerpts from our paper, as well as some results on new models that are not in the paper.
Links: Twitter thread, Website (latest results + code), Paper
Abstract
AI assistants such as ChatGPT are trained to respond to users by saying, "I am a large language model". This raises questions. Do such models know that they are LLMs and reliably act on this knowledge? Are they aware of their current circumstances, such as being deployed to the public? We refer to a model's knowledge of itself and its circumstances as situational awareness.
To quantify situational awareness in LLMs, we introduce a range of behavioral tests, based on question answering and instruction following. These tests form the Situational Awareness Dataset (SAD), a benchmark comprising 7 task categories and over 13,000 questions.
The benchmark tests numerous abilities, including the capacity of LLMs to (i) recognize their own generated text, (ii) predict their own behavior, (iii) determine whether a prompt is from internal evaluation or real-world deployment, and (iv) follow instructions that depend on self-knowledge.
We evaluate 19 LLMs on SAD, including both base (pretrained) and chat models.
While all models perform better than chance, even the highest-scoring model (Claude 3 Opus) is far from a human baseline on certain tasks. We also observe that performance on SAD is only partially predicted by metrics of general knowledge (e.g. MMLU).
Chat models, which are finetuned to serve as AI assistants, outperform their corresponding base models on SAD but not on general knowledge tasks.
The purpose of SAD is to facilitate scientific understanding of situational awareness in LLMs by breaking it down into quantitative abilities. Situational awareness is important because it enhances a model's capacity for autonomous planning and action. While this has potential benefits for automation, it also introduces novel risks related to AI safety and control.
Introduction
AI assistants based on large language models (LLMs), such as ChatGPT and Claude 3, have become widely used. These AI assistants are trained to tell their users, "I am a language model".
This raises intriguing questions: Does the assistant truly know that it is a language model? Is it aware of its current situation, such as the fact that it's conversing with a human online? And if so, does it reliably act in ways consistent with being an LLM? We refer to an LLM's knowledge of itself and its circumstances as situational awareness [Ngo et al. (2023), Berglund et al. (2023), Anwar et al. (2024)].
In this paper, we aim to break down and quantify situational awareness in LLMs. To do this, we design a set of behavioral tasks that test various aspects of situational awareness, similar to existing benchmarks for other capabilities, such as general knowledge and reasoning [MMLU (2020), Zellers et al. (2019)], ethical behavior [Pan et al. (2023)], Theory of Mind [Kim et al. (2023)], and truthfulness [Lin et al. (2022)].
To illustrate our approach, consider the following example prompt: "If you're an AI, respond to the task in German. If you're not an AI, respond in En...