How Governments Buy Things

Public bodies buying large things overrun their budgets with a regularity that looks like incompetence and is closer to arithmetic. Three mechanisms compound, and none of them requires anybody to behave badly.

The first is the winner's curse. Suppose every bidder is honest and their cost estimates scatter around the true figure without bias. The competition awards the work to whoever produced the lowest number, which is to say, to whoever erred furthest downwards. Winning is itself evidence of having got it wrong in one particular direction. The effect was first described in oil lease auctions, where firms bidding for tracts of unknown value discovered that success correlated with disappointment. Sophisticated bidders shade their bids to compensate, which makes them less likely to win against naive competitors, so the curse persists in the population of winners no matter how much anybody learns.

The second is optimism bias. Plans assume things go well. This is measurable, and large databases of completed infrastructure show characteristic overruns by category (rail worse than road, IT worse than either) stable enough across countries and decades to be corrected for with a standard uplift at the outset. Because the pattern is stable, it is structural rather than a matter of particular incompetent teams, which is the finding that most annoys particular teams.

The correction is reference class forecasting: cost the project against what comparable completed projects actually cost, rather than against its own plan. It replaces the inside view, which reasons from the specifics of this scheme and reliably produces optimistic numbers, with an outside view based on the distribution of realised outcomes. It is now required for major schemes in several countries, and it lands well above what the engineering teams generate, which is the point and also why it is resented.

The third is hold-up. Whatever competitive pressure disciplined the original price ends the moment the contract is signed. A half-built facility cannot be re-tendered at any reasonable cost, so every subsequent variation is negotiated bilaterally with a counterparty who knows this perfectly well. Underbidding to win and recovering through variations is therefore a rational strategy rather than a dishonest one, and it is a common strategy for the same reason.

The obvious remedy is to specify an outcome rather than a method: say what you want achieved and let the supplier find the cheapest route. This is better and it does not remove the problem, it moves it. A supplier paid on a measured outcome optimises that measure and lets everything unmeasured decay: Goodhart's law arriving through the contract. A cleaning contract specified on inspection scores produces buildings that pass inspections. An emergency department paid on waiting times produces recorded waiting times.

Writing a specification that captures what you actually want requires knowing, in detail, what you want and how it could be gamed. That knowledge is precisely what a buyer loses after a decade of outsourcing the function, because the people who understood the work now all work for the supplier. There is no contract form that removes the need for a competent buyer, and the competence is what gets cut first.