AF - Can Generalized Adversarial Testing Enable More Rigorous LLM Safety Evals? by Stephen Casper

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Can Generalized Adversarial Testing Enable More Rigorous LLM Safety Evals?, published by Stephen Casper on July 30, 2024 on The AI Alignment Forum.
Thanks to Zora Che, Michael Chen, Andi Peng, Lev McKinney, Bilal Chughtai, Shashwat Goel, Domenic Rosati, and Rohit Gandikota.
TL;DR
In contrast to evaluating AI systems under normal "input-space" attacks, using "generalized," attacks, which allow an attacker to manipulate weights or activations, might be able to help us better evaluate LLMs for risks - even if they are deployed as black boxes. Here, I outline the rationale for "generalized" adversarial testing and overview current work related to it.
See also prior work in
Casper et al. (2024),
Casper et al. (2024), and
Sheshadri et al. (2024).
Even when AI systems perform well in typical circumstances, they sometimes fail in adversarial/anomalous ones. This is a persistent problem.
State-of-the-art AI systems tend to retain undesirable latent capabilities that can pose risks if they resurface. My favorite example of this is the most cliche one
many recent papers have demonstrated diverse attack techniques that can be used to elicit instructions for making a bomb from state-of-the-art LLMs.
There is an emerging consensus that, even when LLMs are fine-tuned to be harmless, they can retain latent harmful capabilities that can and do cause harm when they resurface (
Qi et al., 2024). A growing body of work on red-teaming (
Shayegani et al., 2023,
Carlini et al., 2023,
Geiping et al., 2024,
Longpre et al., 2024), interpretability (
Juneja et al., 2022,
Lubana et al., 2022,
Jain et al., 2023,
Patil et al., 2023,
Prakash et al., 2024,
Lee et al., 2024), representation editing (
Wei et al., 2024,
Schwinn et al., 2024), continual learning (
Dyer et al., 2022,
Cossu et al., 2022,
Li et al., 2022,
Scialom et al., 2022,
Luo et al., 2023,
Kotha et al., 2023,
Shi et al., 2023,
Schwarzchild et al., 2024), and fine-tuning (
Jain et al., 2023,
Yang et al., 2023,
Qi et al., 2023,
Bhardwaj et al., 2023,
Lermen et al., 2023,
Zhan et al., 2023,
Ji et al., 2024,
Hu et al., 2024,
Halawi et al., 2024) suggests that fine-tuning struggles to make fundamental changes to an LLM's inner knowledge and capabilities. For example,
Jain et al. (2023) likened fine-tuning in LLMs to merely modifying a "wrapper" around a stable, general-purpose set of latent capabilities. Even if they are generally inactive, harmful latent capabilities can pose harm if they resurface due to an attack, anomaly, or post-deployment modification (
Hendrycks et al., 2021,
Carlini et al., 2023).
We can frame the problem as such: There are hyper-astronomically many inputs for modern LLMs (e.g. there are vastly more 20-token strings than particles in the observable universe), so we can't brute-force-search over the input space to make sure they are safe.
So unless we are able to make provably safe advanced AI systems (we won't soon and probably never will), there will always be a challenge with ensuring safety - the gap between the set of failure modes that developers identify, and unforeseen ones that they don't.
This is a big challenge because of the inherent unknown-unknown nature of the problem. However, it is possible to try to infer how large this gap might be.
Taking a page from the safety engineering textbook -- when stakes are high, we should train and evaluate LLMs under threats that are at least as strong as, and ideally stronger than, ones that they will face in deployment.
First, imagine that an LLM is going to be deployed open-source (or if it could be leaked). Then, of course, the system's safety depends on what it can be modified to do. So it should be evaluated not as a black-box but as a general asset to malicious users who might enhance it through finetuning or other means. This seems obvious, but there's preced...