Vol. II, No. 42
Covering 15 July - 28 July 2024
Monday, July 29, 2024
Late edition · A Relentless publication
All the fortnight that mattered, in technology and in the world, read next to what we were building at the time.
WASHINGTON

Biden steps aside, and hands his party, and history, to Kamala Harris

After weeks of agony that the debate had begun, the president ended his campaign in a letter on a Sunday afternoon, endorsed his vice-president, and did the rarest thing in politics: he gave up power voluntarily.

President Biden ended his re-election campaign on July 21, in a letter posted to his own account on a quiet Sunday, bowing at last to the tide of doubt that the June debate had unleashed and three weeks of relentless pressure from his own party had made irresistible. Minutes later he endorsed his vice-president, Kamala Harris, who moved with startling speed over the following days to consolidate the support of delegates, donors and elected Democrats, all but securing the nomination before the week was out and transforming, overnight, a race that had felt settled and grim into something volatile and new. The Democratic Party, which a fortnight ago had no plan B, suddenly had one, and a surge of relief and money and energy poured into it.

The manner of it mattered as much as the fact. Biden, 81, did the thing that the powerful almost never do and that this paper has spent the year noting the absence of: he let go. Faced with the evidence that he could not win, and perhaps could not serve, he chose, after visible resistance, to stand down rather than cling on, handing the future to a younger successor. It cost him the thing politicians want most, and he did it, in the end, more gracefully than his long hesitation had suggested he would. Whether Harris can win is the question that will consume the country until November; she inherits a shortened campaign, an energised base, and an opponent, days from his own near-death, riding a wave of his own. But the succession, so long avoided, so painfully forced, had finally happened, and a party that had bet everything on one ageing man got, at the last possible moment, the fresh start it had been too frightened to build for itself.

TECHNOLOGY

A single software update crashes the world for a day

A faulty update from a security company most people had never heard of took down 8.5 million computers at once, grounding flights, freezing hospitals, and darkening broadcasters across the planet. It was not an attack. It was a mistake, deployed everywhere at the speed of the internet.

On the morning of July 19 the world's computers began, all at once, to fail. Airlines grounded their fleets as check-in systems went dark; hospitals cancelled surgeries; banks, broadcasters, supermarkets and emergency lines across continents fell over within the same hour, screens everywhere frozen on the same blue error. The cause was not a cyberattack, which was almost the point. It was a faulty software update, pushed automatically by a cybersecurity company called CrowdStrike to the millions of Windows machines running its Falcon software, a single flawed file that told every one of those computers to crash on boot. Some 8.5 million machines went down, and because the fix required touching each one by hand, the recovery took not minutes but days, airlines cancelling thousands of flights into the following week.

It was the largest IT outage in history, and it was self-inflicted, by an update meant to protect the very systems it destroyed. A company most people had never heard of turned out to sit, invisibly, inside a staggering share of the world's critical infrastructure, with the power to push code directly onto all of it at once, and one bad file exercised that power catastrophically. No enemy did this. No one meant to. A routine update, inadequately tested, deployed to everything simultaneously at the speed the modern software industry treats as a virtue, took down hospitals and airlines and banks across the planet in an afternoon, and the world got a sudden, vertiginous look at how few threads it hangs by, and how tightly they are all tied together.

IN BRIEF

Trump's convention; the Games open on the Seine; Netanyahu in Washington

Donald Trump accepted the Republican nomination in Milwaukee days after the attempt on his life, naming the venture capitalist and senator JD Vance as his running mate and delivering a convention speech that began in the language of unity and returned, at length, to grievance. The Paris Olympics opened on July 26 with an audacious, rain-soaked ceremony staged not in a stadium but along the River Seine, boats carrying the athletes through the heart of the city. And Israel's prime minister Benjamin Netanyahu addressed the US Congress on July 24 to a mix of standing ovations and boycotts, as the Gaza war ground on and ceasefire talks flickered without resolution.

The Column

Normal accidents, and the market that manufactures them

The outage that crashed the world is being blamed on a testing failure. That explanation is comforting and wrong. The catastrophe was not a deviation from how the system works. It was the system working as designed, and the design is one no amount of care will fix.

The software update that took down eight and a half million machines this fortnight, grounding aircraft and freezing hospitals across the planet, is being explained as a quality-control failure: a bad file that better testing would have caught. This is the reassuring story, because it implies a simple remedy, be more careful, and it is wrong, and its wrongness matters, because the remedy it implies will not work and the real one is being ignored. The outage was not a lapse in an otherwise sound system. It was the predictable output of a system built to produce exactly this, and to see why requires setting aside the language of blame and picking up the language of structure.

In 1984 the sociologist Charles Perrow, studying the near-meltdown at Three Mile Island, proposed an idea he called normal accidents. Some systems, he argued, have two properties that together make catastrophe not a risk but a certainty over time. The first is tight coupling: the parts are so interdependent that a failure in one propagates instantly to the others, with no slack, no buffer, no time to intervene. The second is interactive complexity: the parts interact in so many ways that no human can foresee them all, so failures combine in paths no designer anticipated. In such systems, Perrow said, accidents are not aberrations to be engineered away; they are normal, an inherent property of the structure, and the only open questions are when and how large. Perrow was writing about reactors and chemical plants. He did not live to see the modern software supply chain, which is the most tightly coupled and interactively complex system human beings have ever assembled, a structure in which one company can alter millions of machines simultaneously and no one can fully predict what the alteration will touch. By his framework, this fortnight's outage was not a failure of testing. It was a normal accident, as inevitable as rain.

There is a known way to loosen the coupling, and the industry calls it staged rollout: releasing a change to a sliver of machines first, watching, then a few more, so that a bad update poisons a sample rather than the world. It is, precisely, the deliberate reintroduction of slack, and it works. So the real question is not why the outage happened but why the discipline that prevents it is everywhere eroding, and the answer is not carelessness. It is economics, and it is worth stating exactly, because it is the part everyone skips. A firm that stages its rollouts is slower and costlier on every one of the roughly thousand ordinary days; the firm that pushes to everything at once is faster and cheaper on all thousand, and wins the market on all thousand, and loses only on the single tail day, which arrives rarely enough to fall outside the horizon on which markets actually price. So the disciplined firm is punished continuously and the reckless one rewarded continuously, and when the tail day finally comes, the cost of the catastrophe lands not on the firm that removed the slack but on the hospitals and airlines and strangers downstream, who never chose the risk and cannot bill anyone for it. This is not a moral failing of particular engineers. It is a textbook externality: the private incentive to remove slack diverges from the social interest in keeping it, because the firm captures the savings and the world absorbs the disaster.

The strongest objection to all this is the one Silicon Valley would raise, and it is genuinely powerful, so let me put it at full strength. Slack is waste. The relentless stripping of it, the efficiency, the velocity, the continuous deployment, is exactly what let software remake the world and deliver the abundance we now take for granted; the "discipline of slowness" is nostalgia, and the market is right to punish it, because ninety-nine times in a hundred the cautious firm is simply the slower firm, and slower firms deserve to lose. Every word of that is true, and it is why the erosion is so hard to resist. But it contains the error that undoes it. The market prices frequent, small, visible costs superbly and rare, large, externalised costs abominably, and the trouble is that velocity bundles the two together: the same relentless efficiency that is genuine gain on the thousand normal days is uncompensated tail risk on the one catastrophic day, and the market, seeing only the thousand, rewards both as if they were the same thing. It cannot tell the efficiency that creates value from the efficiency that merely borrows against a disaster someone else will pay for, because on every day it can observe, they look identical.

Which is why the fashionable response, be more careful, test more, deploy slower, is worse than useless: it asks individual firms to be virtuous against their own incentives, and virtue does not survive a market that punishes it. You cannot exhort your way out of a structural externality; you have to change the structure, so that the firm which removes the slack bears the cost of the crash. We did this for aircraft, for drugs, for banks, unglamorous liability and regulation and mandated process for the systems whose failure spills onto everyone, and we did it precisely because we learned that "be careful" is not a policy. Until we do the same for the handful of firms whose software can now fail the world in an afternoon, we will keep calling these outages accidents and demanding better testing, and we will keep getting them, on schedule, because they are not accidents and testing is not the lever. The next one is not a risk to be managed. It is an appointment already on the calendar, and we are choosing, by leaving the incentives exactly as they are, to keep it.

The Ledger
AI
The CrowdStrike outage is not an AI story, but it is a devastating preview of one: as more critical systems come to depend on models updated automatically and centrally, the blast radius of a single bad update grows. The industry's "deploy to everything, fast" culture is on a collision course with the fragility it creates.
Data centers & power
The outage showed how concentrated critical infrastructure has quietly become, a single vendor inside a huge share of the world's computers. The same concentration is building in the AI compute layer.
Rates
Soft inflation data all but locked in a September Fed cut, the first of the cycle. The long wait for relief is nearly over.
Real estate
US 30-year mortgage 6.78 percent (July 25), drifting down on firm cut expectations. The thaw approaches as autumn nears.
India tech
The CrowdStrike outage hit Indian airports, banks and offices too, and the recovery leaned heavily on the country's vast IT-services workforce, the people who actually fix millions of machines by hand. A reminder of where the unglamorous, essential labour of keeping the world's computers running actually lives.
What we called wrong
Nothing to retract. The CrowdStrike column above is this paper's oldest argument, the removal of slack in the name of efficiency, applied at last to the author's own trade, and vindicated at planetary scale.
The Back Page

He let go

It is worth pausing, amid the noise, on the rarity of what Joe Biden did this fortnight. He gave up power. Voluntarily, under pressure yes, and after resisting longer than was dignified, but in the end he looked at the evidence, concluded he should not go on, and stood down, handing the future to someone younger. Consider how astonishingly seldom this happens. This paper has spent the year cataloguing the opposite: the strongmen who arrange their own re-coronations, the founders who will not name successors, the ageing leaders who cling until they are carried out, the whole human machinery of not letting go. Power is the thing people fight hardest to keep, that warps judgment, that whispers that only you can do the job and the deluge will follow your departure. And a great many of the people this page has written about, in Moscow and beyond, believed that whisper to the end. Biden, whatever his failures, and the stubborn months of denial were a real failure, did in the end the thing the whisper tells you never to do. He let go. History is not always kind to the ones who hold on too long; it is rarely even asked to judge the ones who step aside in time, because they are so rare. There is no guarantee it changes the election, and it may be remembered as too little, too late. But the act itself, an old man relinquishing the most powerful office on Earth because he judged it was the right thing to do, is worth marking in a year that has shown us, over and over, how few are willing to do it. He let go. Most never do.