On Proof and not-proof, and medicine

There are various kinds of “proof”. Let me first dispose of the proof we find in mathematics. As Feynman pointed out, mathematicians do not need to know what they are talking about. They have axioms, they have rules of inference, and they have proofs. These proofs are in a sense absolutely true. Non-Euclidean geometry is “unreal” in that our world appears to be Euclidean. Nevertheless theorems abound in that space, and they are true given those axioms.

I want to discuss instead the kind of proof we find in science, biology, sociology, and the like, especially as related to health care. Often a result will be considered “significant” because its probability a priori was less than one in twenty. You will see something like p<.05 in journals like Nature. Often the probability is very much smaller, sometimes p<.0001 will appear. This means that the result is truly amazing, akin to being dealt a royal flush in poker.

Now, if I claimed that a poker hand was fixed, the deck stacked, so to speak, and then you were then dealt that royal flush, you might agree with me. We should note here that we don’t really have “proof” in the mathematical, one-hundred point zero percent; we have an unlikely result of probability p where p is very small. Royal flushes do get dealt, just not often.

Suppose instead I said the deck was stacked, and I would get the cards Ace, seven, three, in spades, six of clubs, and two of diamonds. You would not be impressed. But if I did indeed get those cards, you would feel pretty sure I had looked at the deck. While the royal flush is obvious, there are four of them, and only one hand exactly like mine.

The point here is that the unlikelihood is the thing to focus on, not the flashy result.

Where we get into muddy water is with probabilities between .5 and .05. Something that would not be expected half the time, but is more common than once in twenty.

Many studies say “no proven connection” when the probability of the actual result is in this range. This does not mean that the connection is disproven. It merely means that the odds aren’t good enough for the scientist to make a claim of connection.

If you could run many studies and get the same result, the combined probability could enter the realm of “proof”. A single study with this result is not proof, but should not be considered disproof either – it does not prove the negative of its intended statement. Presumably the range .5 to .05 would give the negative a range of .5 to .95, assuming the negative statement is, roughly, “everything else but what we sought can occur”. Again the probability is not in the proof range for the negative either.

This is important because probabilities are part of statistical trials, and drug trials are such. A drug can fail because it does not have long enough odds behind it.

Conversely, a mere probability of .05 indicates decent correlation, perhaps of drug dosage (versus placebo control) with disease remission.

I submit that you do not bet your life repeatedly on odds of one in twenty. If you do this about 14 times in a row you will be, overall, at odds of less than one in two. You might be willing to take one such bet in a life-changing decision, a once-off. If you did it all the time, you would probably lose.

Think about this when you are prescribed something. It may work. It may have side effects. It may be “proved” that the former is likely and the latter unlikely.

Drugs work, or do not work, based on factors we are only beginning to understand. Heredity is part of it. Digestive tract bacteria mix is part of it. Previous experience (exposure to almost anything, not just diseases) is part of it.

You may have been dealt the Ace, seven, three, in spades, six of clubs, and two of diamonds. This may be a good or a bad hand depending on the game you’re playing.

Nobody can prove this in the mathematical sense. Therapies only work in the statistical sense. If you knew about all the trials, including the unpublished ones, you might make a better bet. But it is still a bet, not a certainty.

Being dealt a royal flush could be considered proof, with high likelihood, that you stacked the deck. It is not proof. It is inference from statistics.

Leave a Reply

Your email address will not be published. Required fields are marked *