All Models Are Wrong

George Box wrote that all models are wrong, but some are useful. It is quoted so often that the content has worn off, and the content is worth recovering.

A model is a simplification. That is not a defect to be minimised; it is the entire point. A model containing everything relevant would be the thing itself, and understanding it would be exactly as hard as understanding reality. Every model is therefore wrong by construction, in the specific sense that it leaves things out.

Which turns the evaluation question from "is this model correct?" (the answer is no, always) to three better ones. What does it leave out? Does that matter for the purpose I have? And how would I find out if it did?

A map that omits buildings is useless for finding an address and ideal for planning a route. Nobody calls it wrong; they ask what it is for.

The most common technical failure follows directly from forgetting that simplification is the point.

Given enough parameters, a model can fit any dataset exactly, including its measurement errors and accidents. A polynomial of sufficient degree passes through every point you have. This is overfitting, and the model has then learned the noise as though it were signal.

The symptom is that it performs brilliantly on the data it was fitted to and badly on anything new, which is fatal because new data is the only kind that matters. The standard defence is to hold data back: fit on one portion, evaluate on a portion the model has never seen. Cross-validation is not an optional refinement; without it, goodness of fit tells you almost nothing.

The formal counterpart to Occam's razor lives here. Criteria such as AIC and BIC penalise extra parameters explicitly, trading fit against complexity, and the justification is not a claim that the universe is simple. It is that a more flexible model has more scope to fit noise, so it should have to earn its additional parameters with a proportionate improvement.

Which leads to the error that has cost the most money.

A model that fits historical data well will predict the future well. This is false, and it is false for three separate reasons.

It may fit by capturing noise, as above. It may rely on relationships that held under past conditions and will not hold under new ones. The Lucas critique makes this point about economic relationships that dissolve when a policy tries to exploit them. And it may have been selected from among many candidates on the basis of that same historical data, in which case its fit is partly a selection effect and tells you about the search rather than about the world.

Financial risk models that fitted decades of history have failed catastrophically precisely at the moments they were needed, because the relevant conditions were not in the history.

None of which means a model that predicts poorly is worthless, and this is the part people get wrong in the other direction.

Epidemiological models early in an outbreak predict badly, because the parameters are genuinely unknown; transmissibility, severity, how behaviour will change. Their predictions were widely mocked afterwards for being wrong.

What such a model does supply is the structure of the problem. It shows what difference a given intervention makes under stated assumptions, which quantity the outcome is most sensitive to, and what would have to be true for a scenario to occur. Those are the decision-relevant outputs, and none of them requires the central forecast to be accurate.

A model that identifies which parameter dominates has earned its place even if its number is wrong. The failure is not in the model; it is in reporting a conditional projection as a prediction.

Which gives the right move when two models disagree.

If two models fit the same data equally well and forecast different futures, the disagreement lives in an assumption that the historical data cannot distinguish. Averaging their predictions produces a number that neither model supports and conceals the substantive disagreement.

The useful response is to locate the assumption and design an observation that separates the two. That is the structure of a crucial experiment, and it is the point at which a modelling disagreement becomes an empirical question rather than a matter of taste.