Twenty Years of Keeping Score

In the 1980s Philip Tetlock began asking experts for predictions.

Not vague ones. Specific, dated, checkable claims with probabilities attached: will this country still exist in five years, will this leader still hold office, will inflation exceed this level. Over roughly two decades he collected around 28,000 such forecasts from several hundred people (academics, government analysts, journalists, consultants) and then waited to see what happened.

Average accuracy was barely distinguishable from chance. Tetlock's own comparison was to a dart-throwing chimpanzee, which is the line everybody quotes, and it obscures two findings that matter more.

The first is that fame correlated negatively with accuracy. The experts most in demand by the media were among the least accurate. This is not a puzzle once you see the incentives: what gets you invited back is a confident, simple, dramatic claim, and confident simple dramatic claims about complex systems are usually wrong.

The second is that some forecasters were consistently much better than others, and the difference was not what anybody expected.

It was not subject expertise. It was not years in the field. It was a style of thinking, and Tetlock characterised it using Isaiah Berlin's distinction between the hedgehog, who knows one big thing, and the fox, who knows many small things.

Hedgehogs had a framework (a theory of how the world works) and explained everything through it. They were confident, coherent, quotable, and less accurate. Foxes assembled views from multiple sources, were sceptical of grand theories, held their conclusions loosely, and updated in small increments as evidence arrived.

The foxes won, consistently, across domains.

Tetlock's later Good Judgment Project, run as a forecasting tournament for the US intelligence community, found something more encouraging. Forecasting behaves like a skill with identifiable components.

A short training module (covering base rates, probabilistic reasoning, and common biases) measurably improved accuracy. Working in teams improved it further. And the top few per cent of forecasters, ordinary people identified purely by track record, outperformed intelligence analysts who had access to classified material.

The single most transferable technique is taking the outside view.

Kahneman's illustration is a team he worked with on a textbook, who estimated they would finish in about two years. He then asked a member who had seen similar projects how long comparable efforts had taken. The answer was seven to ten years, and around 40% never finished at all.

The team had been taking the inside view: building an estimate from the specifics of their own project, their own plan, their own capabilities. The outside view asks how often things of this kind go how, before considering any specifics at all. It is systematically more accurate and psychologically unsatisfying, because it treats your situation as an instance of a category rather than as the particular thing it obviously is.

Their book took eight years.

All of this depends on one methodological insistence, which is where most of the value actually came from.

A prediction that something may happen soon cannot be checked, and a very large proportion of public forecasting is phrased exactly that way. Scoring rules such as the Brier score require a probability and a date, and reward being both accurate and appropriately confident, hedging everything at 50% scores badly too.

Once predictions are recorded in a gradeable form, accountability becomes possible for the first time, and Tetlock's most important contribution may be less the discovery that experts forecast poorly than the demonstration that almost nobody was keeping score.

Which gives a practical standard for anybody evaluating a commentator.

A single wrong prediction proves very little. Good probabilistic judgement is wrong some of the time by construction: a forecaster who says 70% should be wrong three times in ten, and one who is never wrong was not saying 70%.

What is diagnostic is calibration across many dated predictions, and the near-total absence of such records is itself the finding. When someone explains why their reasoning was sound despite the outcome, the question is not whether the explanation is plausible. It is whether there is a record, and what it says.