The Reproducibility Project

In 2015 the Open Science Collaboration published the results of an unusual project. Two hundred and seventy researchers had taken one hundred published psychology studies (from good journals, most with statistically significant results) and attempted to repeat them, with larger samples and, wherever possible, the original authors' cooperation and materials.

Around thirty-six per cent produced a statistically significant result in the same direction. Average effect sizes were roughly half the published ones.

The number is often quoted as though it settles something, and it does not: some replications were imperfect, some effects genuinely depend on context, and a single failed replication is weak evidence about any particular finding. What the project established was not that specific results were false. It was that the published literature as a body was not a reliable guide to what would happen if you did the experiment again, and that is a different and more serious problem.

The causes turned out to be structural rather than scandalous, which is why the episode is worth understanding rather than merely deploring.

Start with publication bias. Journals publish positive findings. A study that finds nothing is harder to publish, less cited, and often never written up: the file drawer. So if twenty labs test an effect that does not exist, and each uses the conventional five per cent threshold, one will get a significant result by chance, and that one gets published. The literature then contains a finding supported by one study and contradicted by nineteen that nobody can read.

Then there is the subtler mechanism, and the one that implicated almost everybody. Analysing data involves decisions: which outliers to exclude, which covariates to include, when to stop collecting, which of several measures to report. Each decision is defensible on its own. Made after seeing the data, and chosen with an eye to which version comes out significant, they multiply into what Andrew Gelman called the garden of forking paths: a space of analyses large enough that something will clear the threshold. The researcher need not be dishonest, and typically was not. They simply did what looked reasonable at each step, and never saw the other paths they might have taken.

Add low statistical power. Small samples produce noisy estimates, which means the significant results from underpowered studies are the ones where noise happened to run large, so the published effect is inflated, sometimes enormously. A field that runs many small studies and publishes only the significant ones will produce a literature of exaggerations even if every individual researcher behaves impeccably.

What makes this episode creditable rather than shameful is the response. Preregistration (writing down the hypothesis and the full analysis plan before collecting data, and depositing it publicly) closes the forking paths by making the choices auditable. Registered reports go further: journals review and accept the design before results exist, so acceptance cannot depend on what is found. Sample sizes have risen sharply. Data and materials are increasingly posted. Direct replications, once treated as unpublishable and slightly rude, now appear in the same journals as the originals.

None of this is unique to psychology. Cancer biology's replication rates are comparable or worse, economics and medicine have run their own versions of the exercise, and the incentive structure that produced the problem (publish, be novel, be positive) is universal across research. Psychology's distinction is that it went looking first and published what it found about itself.

Which suggests the right way to hold the whole affair. A field that runs a large, public, adversarial audit of its own canon and reports a thirty-six per cent hit rate has not thereby shown itself to be untrustworthy. It has demonstrated the one capability that makes a discipline worth trusting at all.