Significant, and Other Misread Words

In 2009 a neuroscientist named Craig Bennett put a dead Atlantic salmon into an fMRI scanner. The fish was shown photographs of humans in social situations and asked to determine what emotion each person was experiencing.

The analysis found a cluster of voxels in the salmon's brain cavity showing a statistically significant response.

The point of the exercise was not that fMRI does not work. It was that an fMRI scan involves tens of thousands of separate statistical tests, one per voxel, and at a five per cent threshold a certain number will come out significant on noise alone. Without correction for multiple comparisons, you can find brain activity in a dead fish, and the paper was a way of saying so that nobody could ignore.

Which is the best place to start with p-values, because the multiple comparisons problem is a direct consequence of what a p-value is.

A p-value is the probability of getting data at least this extreme if there were no real effect. That is it. It is a statement about the probability of the data given a hypothesis, and people almost universally read it as the probability of the hypothesis given the data: a completely different quantity, which cannot be computed without knowing how likely the hypothesis was to begin with. "There is a three per cent chance this is a fluke" is the standard misreading, and it is wrong in a way that matters: if you are testing an implausible hypothesis, most of your significant results will be flukes no matter how small the p-values are.

The threshold of 0.05 has no principled basis at all. Fisher suggested it as a convenience and said explicitly that it was a matter of judgement. It has since become a publication criterion, which is how you get a literature sorted by whether a number fell on one side of an arbitrary line.

And it says nothing about size. Significance depends heavily on sample size, so with a million participants a difference of no practical consequence will be highly significant, and with thirty a large real effect may miss the threshold entirely. The useful report is an effect size with a confidence interval, which tells you how big the effect might plausibly be and how precisely it has been pinned down, and which, unlike a p-value, can tell you that a study was uninformative rather than negative.

The same confusion between relative and absolute runs through medical reporting. A drug that halves your risk sounds like the same thing whether your risk was one in ten or one in ten thousand, and it is not. Relative risk reduction is scale-free and therefore uninformative alone, which is precisely why press releases prefer it. Absolute risk reduction gives the actual change in your chances. The number needed to treat (one divided by the absolute reduction) tells you how many people must take the drug for one to benefit, and it is the number that makes a decision possible. The same honest trial can be described as a fifty per cent reduction or as two people in ten thousand.

Then there is the effect that manufactures findings out of nothing, and it is worth knowing by name because it is everywhere.

Francis Galton noticed that tall fathers had sons who were tall but less so, and called it regression to mediocrity. The mechanism is simply that any measurement is partly signal and partly noise, so an extreme measurement is likely to have had luck pushing it outward, and the luck will not repeat. Extreme values are followed by less extreme ones.

Kahneman's illustration is the Israeli flight instructors, who told him that praising a cadet after an exceptional landing was followed by a worse one, and criticism after a bad landing by improvement, and concluded that criticism works and praise backfires. Both patterns are exactly what regression predicts with no effect of the instruction at all. An exceptional landing was partly luck; the next one will be closer to the cadet's average whatever anyone says.

The general form: any intervention applied when things are at their worst will appear to work. Hospitals that introduce a programme after a bad year, schools that are inspected after poor results, a back treatment sought at peak pain; all will show improvement, and none of it is evidence. The only way to separate the treatment from the regression is a control group measured at the same point.

Finally, the question that should follow every correlation, which is: which way does the arrow point, and is there a third thing?

A study finds that hospitals treating more patients have better survival rates, and concludes that practice makes perfect. Perhaps. Or better hospitals attract more patients, which is the same correlation with the arrow reversed. Or specialist centres take the hardest cases, which would understate the effect, or refer them away, which would overstate it. All three are live, the volume-outcome relationship is genuinely debated in the literature, and settling it requires finding situations where volume changed for reasons unrelated to quality: a nearby unit closing, say. Which is what the good studies do, and it is a great deal more work than running a correlation.