Suppose something important will happen at a certain unknown point. As someone approaches that point, you might be tempted to warn that the thing will happen. If you’re being appropriately cautious, you’ll warn about it before it happens. Then your warning will be wrong. As things continue to progress, you may continue your warnings, and you’ll be wrong each time. Then people will laugh at you and dismiss your predictions, since you were always wrong before. Then the thing will happen and they’ll be unprepared.

Toy example: suppose you’re a doctor. Your patient wants to try a new experimental drug, 100 mg. You say “Don’t do it, we don’t know if it’s safe”. They do it anyway and it’s fine. You say “I guess 100 mg was safe, but don’t go above that.” They try 250 mg and it’s fine. You say “I guess 250 mg was safe, but don’t go above that.” They try 500 mg and it’s fine. You say “I guess 500 mg was safe, but don’t go above that.”

They say “Haha, as if I would listen to you! First you said it might not be safe at all, but you were wrong. Then you said it might not be safe at 250 mg, but you were wrong. Then you said it might not be safe at 500 mg, but you were wrong. At this point I know you’re a fraud! Stop lecturing me!” Then they try 1000 mg and they die.

The lesson is: “maybe this thing that will happen eventually will happen now” doesn’t count as a failed prediction.

I’ve noticed this in a few places recently.

https://www.astralcodexten.com/p/against-the-generalized-anti-caution