For twenty years the British Post Office prosecuted, bankrupted, imprisoned, and in some cases drove to suicide hundreds of its own subpostmasters, ordinary people running village post offices, because a computer system called Horizon reported shortfalls in their accounts, money that was never actually missing but that the software, riddled with faults, invented, and it took a television drama, this fortnight, to force a nation to believe the victims it had spent two decades disbelieving. The scandal is being told, rightly, as a story of institutional cruelty and cover-up. But underneath the cruelty is a structural failure that I encounter, in smaller and less murderous forms, in nearly every organisation that runs on software, and I want to name it precisely, because it is spreading and it is about to get much worse.
Lisanne Bainbridge wrote a paper in 1983 called "Ironies of Automation" that everyone who builds these systems should be made to read, in which she identified a perverse dynamic: as we automate a process and it becomes more reliable, the humans left to supervise it lose the practice, the context, and eventually the standing to challenge it, so that precisely as the system's rare failures become more consequential, the humans become less capable of catching them, and more disposed to defer to a machine that is usually right. This is automation bias, the well-documented human tendency to trust the automated output over our own judgment and over other humans, and it grows stronger the more reliable the system usually is, because every day the computer is right trains the operator to assume it is right, until the day it is catastrophically wrong and there is no longer anyone in the loop with the confidence, the knowledge, or the authority to say so. Horizon is Bainbridge's irony at industrial scale and across two decades: a system trusted so completely that the word of the machine outweighed the word, and the character, and the desperate protestations, of hundreds of people who had kept honest books their whole lives.
Let me put the counterargument fairly, because it is not stupid and it is in fact usually correct, which is exactly what makes the trap so deep. Computers are, at arithmetic and record-keeping, genuinely more reliable than humans; the whole reason we automate accounting is that human bookkeeping is error-prone and slow and inconsistent, and an institution that overruled its accounting system every time a person objected would be paralysed, defrauded, and correct far less often than one that trusted the system. In the overwhelming majority of cases, believing the computer over the individual's protest is the right call, statistically, and an organisation cannot function if every automated result is up for negotiation with whoever it inconveniences. This is true, and it is the reason the Post Office's posture was not obviously insane from the inside, and that is the whole horror of it: the failure was not a crude refusal to think, it was the over-application of a heuristic that is usually right, pushed past the point where anyone retained the capacity to notice it had become catastrophically wrong. The danger is never that the machine is trusted; it is that it is trusted structurally, with no preserved mechanism, no empowered human, no institutional will, to catch the case where the usually-right system is this time destroying the innocent.
Because that is where the real fault lies, and it is not in the software, which was merely, banally, buggy, as all software is. The fault is that the institution arranged itself so that the machine's output was effectively unchallengeable: it stripped the local humans of the authority to dispute it, gave them no window into how it worked, structured the incentives so that admitting the system could be wrong threatened the whole edifice and the careers atop it, and thereby removed, one by one, every point at which a human being might have interposed judgment between the machine's error and a person's ruin. The computer said money was missing, and the institution had, over years, dismantled every position from which anyone could effectively say: the computer is wrong, and this person is telling the truth, and I have the standing and the duty to act on that. There was no one left to disagree, not because the people were absent but because the structure had disempowered them, and a machine that cannot be contradicted is not a tool. It is an oracle, and an institution that has built an oracle has abolished responsibility, because responsibility requires a human who can be held to account, and no one can be held to account for deferring to the computer.
I write this now because we are about to do it everywhere, at a scale that will make Horizon look small. We are wiring automated judgment into hiring, lending, policing, medicine, welfare, the allocation of life chances, and we are doing it with the same reassurance that the system is usually right, which is usually true, and with the same quiet erosion of the human positions from which the system could be challenged, and with the same institutional preference for the machine's clean answer over the messy, inconvenient, expensive protest of the individual it has wronged. The lesson of the Post Office is not that we should distrust computers, which is neither possible nor wise. It is that a system, however reliable, must never be allowed to become unchallengeable, that there must always be a human with the knowledge to understand it, the authority to overrule it, and the incentive to listen to the person it is about to destroy, and that building that human into the loop, and protecting them, is not a brake on efficiency but the whole difference between a tool and an oracle. Britain built an oracle in its post offices and fed hundreds of innocent people to it over twenty years, and the machine was only wrong. It was the people who arranged for no one to be left to disagree who were guilty, and we are, right now, arranging the same thing, on a hundred times the scale, and calling it progress.