
“Thoughts on “safety bad bc warning shots good”” by Cleo Nardo
Suppose there was a button which made the current generation of AIs more safe, i.e. more aligned and less capable of subverting control measures. The button has no other effects — in particular, the button wouldn’t make future generations more or less safe. Should you press the button?
Maybe you shouldn’t, because of the Warning Shot Argument:
- Future agents contribute almost all the risk of catastrophe, compared with current agents.
- Pressing the button makes current agents safer.
- If current agents are safer, then they are less likely to cause non-catastrophic harms, e.g. cybercrime.
- If current agents don’t cause non-catastrophic harms, then people will be less aware about catastrophic risks, e.g. lab leadership, governments, legislators, international community, lab employees, general public, media, academia, etc.
- If people are less aware about catastrophic risks, then they won’t take drastic action to reduce catastrophic risk.
- So future agents contribute more catastrophic risk.
- Therefore pressing the button would increase catastrophic risk.
This might seem very abstract, but consider a concrete example. Suppose you’d pressed the button to make OpenAI's HPIM more aligned and less capable to subvert control measures. Then the HuggingFace incident would never have happened. And that probably would’ve increased catastrophic [...]
---
Outline:
(01:48) Is this self-defeating?
(03:07) Other considerations
---
First published:
November 17th, 2026
Source:
https://www.lesswrong.com/posts/qJrhEoNbzKRQYhybh/thoughts-on-safety-bad-bc-warning-shots-good
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Altri episodi di "LessWrong (30+ Karma)"



Non perdere nemmeno un episodio di “LessWrong (30+ Karma)”. Iscriviti all'app gratuita GetPodcast.








