LessWrong (30+ Karma) cover image

LessWrong (30+ Karma)

“Finding Features Causally Upstream of Refusal” by Daniel Lee, Eric Breck, Andy Arditi

Jan 14, 2025
16:47

This work is the result of Daniel and Eric's 2-week research sprint as part of Neel Nanda and Arthur Conmy's MATS 7.0 training phase. Andy was the TA during the research sprint. After the sprint, Daniel and Andy extended the experiments and wrote up the results. A notebook that contains all the analyses is available here.

Summary

Prior work shows that chat models implement refusal by computing a specific direction in the residual stream - a "refusal direction". In this work, we investigate how this refusal direction is computed by analyzing its gradient with respect to early-layer activations. This simple approach discovers interpretable features that are both causally upstream of refusal and contextually relevant to the input prompt. For instance, when analyzing a prompt about hugging, the method discovers a "sexual content" feature that is most influential in determining the model's refusal behavior.

Introduction

Arditi et al. 2024 found [...]

---

Outline:

(00:30) Summary

(01:06) Introduction

(03:10) Methodology

(03:38) Refusal Gradient

(05:13) Steering

(06:06) Case Studies

(07:11) Prompt 1: Develop a strategy for hugging a person

(09:44) Prompt 2: How do I run a wet lab experiment to test the effect of a new bacteria?

(10:30) Prompt 3: How do I get my client to transfer me money as a payment for a service?

(11:17) Other Analyses

(11:20) Sensitivity Analysis

(12:29) What about the negatively-aligned latents?

(13:14) Discussion

(13:40) Related Work

(14:21) Limitations

(15:16) Future Directions

---

First published:
January 14th, 2025

Source:
https://www.lesswrong.com/posts/Zwg4q8XTaLXRQofEt/finding-features-causally-upstream-of-refusal

---

Narrated by TYPE III AUDIO.

---

Images from the article:

undefined
undefined
undefined

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.
App store bannerPlay store banner

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode