I’ll add some stuff from Nature on this later, maybe.
Probability and experiments are often misunderstood.
Here is a fun example we should all read, and laugh at, and then laugh at ourselves.
Trying 20 types of candies, one in 20 comes up with a positive result. For that one, P<.05.
Since it is one of twenty, that shouldn’t be seen as ‘proof’ but the P value will be seen as such.
The Nature article insists that the pre-experiment known possibilities, including for false positives, means that many of our ‘accepted’ results are, er, well, not that acceptable.