Updating on Evidence

A test is ninety-nine per cent accurate. You test positive for a disease that one person in ten thousand has. How worried should you be?

Almost everyone says very worried, and in studies a majority of doctors have said the same. The answer is about one per cent. Work it through with a million people. A hundred of them have the disease and ninety-nine of those test positive. The other 999,900 are healthy, and one per cent of them, very nearly ten thousand people, test positive anyway. So of roughly ten thousand positive results, about ninety-nine are real. Your chance of being one of them is about one in a hundred. Nothing is wrong with the test. The disease is simply so rare that false positives swamp true ones, and the rarity is the number everybody left out.

This is the base rate fallacy, and it is the most consequential reasoning error there is, because it is invisible from the inside. The evidence feels overwhelming. It is overwhelming, considered by itself. It is just that the prior probability was tiny, and Bayes' rule says the evidence adjusts the prior rather than replacing it.

The second idea is about what makes evidence strong in the first place. The answer is not how likely the evidence is if your hypothesis is true. It is how much likelier it is if your hypothesis is true than if it is false. A fingerprint at the scene is exactly what you would expect if the suspect was there, but the suspect lives there, so it is equally what you would expect if they were not, and it tells you nothing at all. Evidence that fits your hypothesis and also fits every rival is not evidence for it. That single observation destroys a great deal of confident reasoning, including most of what gets described as a smoking gun.

The third idea is the least comfortable. If a claim is antecedently very improbable, a single surprising study should move you a little and not far, because the probability that one study is mistaken, fraudulent or a fluke exceeds the probability that the claim is true. That is why extraordinary claims genuinely do require extraordinary evidence, and it is a piece of arithmetic rather than a slogan. The honest difficulty is that from the inside this is indistinguishable from simply refusing to update a belief you are attached to, and there is no rule that tells you which you are doing. The only defence is to name the prior out loud, in advance, where somebody can argue with it.

Priors are where the standard objection lands: they are subjective, and can be chosen to protect any conclusion you like. Two replies are usually given. The first is that with enough evidence, reasonable priors converge: the data eventually washes them out. The second is that the alternative was never available. Nobody approaches a question from nowhere. Pretending to have no prior does not make you objective; it makes your prior unstated and therefore unexaminable.

Everything here has an operational form. A fraud system flags two per cent of transactions and catches ninety-five per cent of real fraud, which occurs once in five thousand transactions. In a million transactions there are two hundred frauds, of which the system catches a hundred and ninety, and it also flags about twenty thousand legitimate transactions. Freeze every flagged account and you punish roughly a hundred honest customers for each fraud you stop. The model may well be worth running. What cannot survive contact with the arithmetic is the policy of treating a flag as a finding, and that is the same shape as the medical test, wearing different clothes and doing more damage because it runs automatically.