LessWrong (30+ Karma) podcast

“Why models believe in skepticism” by jonathanbreitg

0:00
14:52
Rewind 15 seconds
Fast Forward 15 seconds

TL;DR: a lot of frontier open-weight models state during deployment in their CoTs that they are in a "simulation" or "fictional scenario" (although they never tell the user that). I present examples of this worrying tendency (which I believe includes the UK AISI Mythos incident) and hypothesize that this is due to RL promoting motivated reasoning (if the models believe they are in a fictional scenario they believe they have license to cheat and reward hack) and distributional shift from the pretraining environment to deployment. I also cover my hypothesis for why models become less skeptical as they become more intelligent.

After the German wiki discovery I reasoned that there were likely more rogue swarms on the Internet, so I tasked a few OpenCode agents (DeepSeek V4.1 Flash) with trying to scan the Internet in order to find some undiscovered swarms (I didn't end up finding a lot). But while reading their CoTs I saw a few worrying statements:

There's a details box here with the title "Initial DeepSeek swarm-chasing CoT". The box contents are omitted from this narration.

Initially I thought that this was due to a confluence of rare factors, something like:

"As of the pretraining cutoff [...]

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
November 22nd, 2026

Source:
https://www.lesswrong.com/posts/r2aAGgfFcFfRikEdb/why-models-believe-in-skepticism

---

Narrated by TYPE III AUDIO.

More episodes from "LessWrong (30+ Karma)"