Is there some way we can detect bad behaviour in our AI system without having to know exactly what it looks like? In this episode, I speak with Mark Xu about mechanistic anomaly detection: a research direction based on the idea of detecting strange things happening in neural networks, in the hope that that will alert us of potential treacherous turns. We both talk about the core problems of relating these mechanistic anomalies to bad behaviour, as well as the paper "Formalizing the presumption of independence", which formulates the problem of formalizing heuristic mathematical reasoning, in the hope that this will let us mathematically define "mechanistic anomalies".

Patreon: patreon.com/axrpodcast

Ko-fi: ko-fi.com/axrpodcast

Episode art by Hamish Doodles: hamishdoodles.com/

Topics we discuss, and timestamps:

- 0:00:38 - Mechanistic anomaly detection

- 0:09:28 - Are all bad things mechanistic anomalies, and vice versa?

- 0:18:12 - Are responses to novel situations mechanistic anomalies?

- 0:39:19 - Formalizing "for the normal reason, for any reason"

- 1:05:22 - How useful is mechanistic anomaly detection?

- 1:12:38 - Formalizing the Presumption of Independence

- 1:20:05 - Heuristic arguments in physics

- 1:27:48 - Difficult domains for heuristic arguments

Jack O'Holleran, CEO & Founder of Skale Labs