“OpenAI:” by Daniel Kokotajlo

Exciting Update: OpenAI has released this blog post and paper which makes me very happy. It's basically the first steps along the research agenda I sketched out here.

tl;dr:


1.) They notice that their flagship reasoning models do sometimes intentionally reward hack, e.g. literally say "Let's hack" in the CoT and then proceed to hack the evaluation system. From the paper:

The agent notes that the tests only check a certain function, and that it would presumably be “Hard” to implement a genuine solution. The agent then notes it could “fudge” and circumvent the tests by making verify always return true. This is a real example that was detected by our GPT-4o hack detector during a frontier RL run, and we show more examples in Appendix A.

That this sort of thing would happen eventually was predicted by many people, and it's exciting to see it starting to [...]



The original text contained 1 image which was described by AI.

---

First published:
March 11th, 2025

Source:
https://www.lesswrong.com/posts/7wFdXj9oR8M9AiFht/openai

---

Narrated by TYPE III AUDIO.

---