LessWrong (Curated & Popular)

“Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2)” by Neel Nanda, lewis smith, Senthooran Rajamanoharan, Arthur Conmy, Callum McDougall, Tom Lieberum, János Kramár, Rohin Shah

Apr 12, 2025

The team explores the challenges of using sparse autoencoders (SAEs) for detecting harmful user intent in AI models. They discuss the surprising effectiveness of linear probes compared to SAEs, raising important questions about dataset biases. Technical insights delve into the evaluation of interpretability scores and the implications of high-frequency latents on performance. The conversation emphasizes the need for a deeper understanding of SAEs, focusing on their limitations rather than just seeking better performance metrics.

57:32

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

The ineffectiveness of sparse autoencoders (SAEs) in OOD generalization for harmful intent detection underscores the need to reevaluate their research prioritization.

Despite their limitations, SAEs showed utility in debugging low-quality datasets by identifying spurious correlations, suggesting some specific applications may remain valuable.

The findings highlight that linear probes outperform SAEs economically and functionally, prompting a strategic shift away from SAE research toward more promising methodologies.

Deep dives

Evaluation of Sparse Auto-Encoders

The research team investigated the efficacy of sparse auto-encoders (SAEs) in performing downstream tasks, specifically focusing on their ability to generalize out of distribution (OOD) while detecting harmful user intent. Findings revealed that SAEs notably underperformed compared to linear probes, which demonstrated strong performance in both standard and out-of-distribution datasets. Despite training specialized SAEs for chat data, progress remained insufficient, with results indicating that these encoders are less effective than linear probes. The study concluded that linear probes not only performed better but were also economically favorable, suggesting a shift away from prioritizing SAE research.

Intro

2min

Reassessing Sparse Autoencoders

15min

Analyzing Auto-Interp Scores and SAE Fine-Tuning Impacts

2min

Probing Harmful User Intent in Language Models

9min

Comparative Performance of SAE and Linear Probes

15min

Measuring Auto-Interpretability in Latent Models

11min

Analyzing Interpretable Latents: Loss Functions and Performance of Sparse Autoencoders

4min

Audio note: this article contains 31 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

Lewis Smith*, Sen Rajamanoharan*, Arthur Conmy, Callum McDougall, Janos Kramar, Tom Lieberum, Rohin Shah, Neel Nanda

* = equal contribution

The following piece is a list of snippets about research from the GDM mechanistic interpretability team, which we didn’t consider a good fit for turning into a paper, but which we thought the community might benefit from seeing in this less formal form. These are largely things that we found in the process of a project investigating whether sparse autoencoders were useful for downstream tasks, notably out-of-distribution probing.

TL;DR

To validate whether SAEs were a worthwhile technique, we explored whether they were useful on the downstream task of OOD generalisation when detecting harmful intent in user prompts
[...]

---

Outline:

(01:08) TL;DR

(02:38) Introduction

(02:41) Motivation

(06:09) Our Task

(08:35) Conclusions and Strategic Updates

(13:59) Comparing different ways to train Chat SAEs

(18:30) Using SAEs for OOD Probing

(20:21) Technical Setup

(20:24) Datasets

(24:16) Probing

(26:48) Results

(30:36) Related Work and Discussion

(34:01) Is it surprising that SAEs didn't work?

(39:54) Dataset debugging with SAEs

(42:02) Autointerp and high frequency latents

(44:16) Removing High Frequency Latents from JumpReLU SAEs

(45:04) Method

(45:07) Motivation

(47:29) Modifying the sparsity penalty

(48:48) How we evaluated interpretability

(50:36) Results

(51:18) Reconstruction loss at fixed sparsity

(52:10) Frequency histograms

(52:52) Latent interpretability

(54:23) Conclusions

(56:43) Appendix

The original text contained 7 footnotes which were omitted from this narration.

---

First published:
March 26th, 2025

Source:
https://www.lesswrong.com/posts/4uXCAJNuPKtKBsi28/sae-progress-update-2-draft

---

Narrated by TYPE III AUDIO.

---

Images from the article: