Vol. I, No. 1
Covering 19 December 2022 - 1 January 2023
Monday, January 2, 2023
Late edition · A Relentless publication
All the fortnight that mattered, in technology and in the world, read next to what we were building at the time.
AVIATION

Southwest cancels more than 15,000 flights as its own scheduling system buckles under a storm

The winter storm hit every airline. Only one of them fell over, and it was the one that had been told for years its crew software could not cope.

Southwest Airlines spent the last ten days of the year doing something no large American carrier had done in the modern era: cancelling most of its schedule for the better part of a week while its competitors flew. From December 22 through December 29 the airline scrubbed more than 15,000 flights, on some days close to two thirds of everything it had planned. Bags piled up in terminals in Denver, Chicago Midway, Baltimore and Nashville. Families slept on floors. Crews sat in hotel rooms, legally rested and ready to work, and could not get through to anyone who could tell them where to go.

Winter Storm Elliott was the trigger. It brought sub-zero wind chills to Texas, buried Buffalo, New York, and killed dozens of people across the country, most of them in Erie County. Every airline cancelled flights on December 22 and 23. But by December 26 the rest of the industry was running near normal, and Southwest was still cancelling more than 2,500 flights a day.

The difference, by the airline's own admission, was software. Southwest runs a point-to-point network rather than a hub-and-spoke one, which means crews and planes are scattered across the map rather than cycling through a few fortresses. When the storm broke that pattern, the system the airline uses to reassign crews, an ageing package the company calls SkySolver, could not keep up with the number of changes. Dispatchers fell back to phone calls. Pilots reported hold times of hours. The airline's chief operating officer, Andrew Watterson, told employees that the tools had not been able to keep up with the volume of changes. The pilots' union had said much the same thing publicly for years.

The Department of Transportation said it would examine whether the cancellations were "controllable" and whether the airline was meeting its obligations on refunds and expenses. Secretary Pete Buttigieg called the meltdown "unacceptable" in a series of television interviews and said his department would hold the airline to its commitments. Southwest's chief executive, Bob Jordan, apologised in a video message and said the airline would refund tickets and cover reasonable expenses.

The airline had not put a number on the cost by New Year's Day. Analysts were talking about hundreds of millions of dollars in refunds, vouchers and lost revenue before counting the reputational bill. The stock fell more than 7 percent between the start of the storm and the end of the year.

What made the episode notable was not the storm. It was that a company with a market value north of $20 billion and a famously loyal customer base had been running the middle of its operation on a system that everyone inside knew would break under exactly these conditions, and had chosen, year after year, to spend the money elsewhere.

TECHNOLOGY

Google declares a "code red" as ChatGPT reaches a million users

A month-old chatbot from a research lab has the largest search company in the world rearranging its product plans.

OpenAI's ChatGPT, released on November 30 as what the company called a "research preview," reached a million users in five days and kept growing through December. By the middle of the month it was the thing people were showing each other on their phones: writing cover letters, explaining code, drafting emails in the style of a pirate.

On December 21 The New York Times reported that Google's management had declared a "code red" internally over the tool, with chief executive Sundar Pichai redirecting teams across the company to work on AI products. The concern was direct. If people start asking a chatbot instead of a search box, the advertising model that produces most of Alphabet's revenue is exposed.

Google has been building large language models for years and has one, called LaMDA, that its own engineers say is at least as capable. It has held back from releasing it as a product, citing the reputational risk of a system that states falsehoods with confidence. ChatGPT does that too. The difference is that OpenAI shipped it anyway.

Microsoft, which invested $1 billion in OpenAI in 2019, has said nothing publicly about what it plans to do with the technology.

CRYPTO

Bankman-Fried extradited, freed on $250 million bond as two lieutenants plead guilty

Sam Bankman-Fried, the founder of the collapsed cryptocurrency exchange FTX, was flown from the Bahamas to New York on December 21 and released the next day on a $250 million bond, secured against his parents' home in Palo Alto, California. He will live there under electronic monitoring while awaiting trial on eight counts including wire fraud and conspiracy to commit money laundering.

Hours before his court appearance, prosecutors announced that Caroline Ellison, who ran the trading firm Alameda Research, and Gary Wang, an FTX co-founder, had pleaded guilty and were cooperating. FTX customers are owed billions of dollars that the company does not have.

CHINA

Beijing abandons quarantine for arrivals as infections sweep the country

China said on December 26 that it will end mandatory quarantine for inbound travellers on January 8, the last major piece of the "zero-Covid" system it dismantled in the first week of December after nationwide protests. Cases have surged since the restrictions were lifted. Hospitals in Beijing and Shanghai reported full wards; funeral homes reported queues. The official death count, which requires a positive test and respiratory failure, remained in the single digits per day and was widely disbelieved. Several countries, including the United States, Japan, Italy and India, imposed testing requirements on travellers from China.

WASHINGTON

Zelensky addresses Congress; $1.7 trillion spending bill passes

Ukraine's president, Volodymyr Zelensky, made his first trip abroad since the Russian invasion in February, flying to Washington on December 21 to meet President Biden and address a joint session of Congress. "Ukraine is alive and kicking," he told lawmakers, and asked for more air defence. The administration announced it would send a Patriot missile battery. Two days later Congress passed a $1.7 trillion omnibus spending bill that includes roughly $45 billion for Ukraine; the president signed it on December 29.

MARKETS

Bank of Japan surprises with a shift on yields; Tesla ends its worst year

The Bank of Japan, the last major central bank still holding rates below zero, widened the band in which it lets 10-year government bond yields trade, to half a percentage point, on December 20. Markets read it as the first crack in a decade of ultra-loose policy; the yen jumped and government bond yields rose worldwide. Separately, Tesla finished 2022 down about 65 percent, its worst year as a public company, as investors questioned demand for its cars and the attention of its chief executive, who spent the autumn buying and running Twitter.

TWITTER

Musk says he will step down as chief executive, once he finds someone "foolish enough"

Elon Musk asked Twitter users on December 18 whether he should step down as head of the company. Some 17.5 million votes were cast; 57.5 percent said yes. On December 20 he said he would resign as chief executive "as soon as I find someone foolish enough to take the job," and would then run the software and server teams. The poll followed a week in which the company suspended and then reinstated the accounts of several journalists, and Musk banned and then unbanned links to rival social networks.

PRIVACY

Epic Games to pay $520 million to settle FTC charges over children's data and "dark patterns"

The maker of Fortnite agreed on December 19 to pay a record $275 million penalty for collecting personal information from children under 13 without parental consent, and a further $245 million in refunds to customers who were tricked into unwanted purchases by confusing checkout designs. The Federal Trade Commission called it the largest penalty ever for a violation of an FTC rule.

DEATHS

Benedict XVI, Pelé, Barbara Walters, Vivienne Westwood

Pope emeritus Benedict XVI, the first pontiff to resign in six centuries, died on December 31 at 95. Pelé, the only man to win three World Cups, died on December 29 at 82. Barbara Walters, who broke the network anchor desk open for women, died on December 30 at 93. The designer Vivienne Westwood died on December 29 at 81. On January 1, Croatia adopted the euro and joined the Schengen area, and Luiz Inácio Lula da Silva was sworn in as president of Brazil for a third time, two days after his predecessor flew to Florida.

The Column

The people who keep the lights on

A great airline was brought to its knees by software everyone inside it knew would fail. The meltdown is being told as a story about a winter storm. It was a story about a bill coming due, one that had been deferred, deliberately and profitably, for twenty years.

Southwest Airlines cancelled more than fifteen thousand flights over the holidays this fortnight, stranding a million people, when a winter storm pushed its decades-old crew-scheduling system past the point where it could track where its own pilots and flight attendants were. The story is being told as a weather story, an act of God that overwhelmed an unlucky airline, and that telling is convenient for everyone, because a storm is nobody's fault. But the people inside that airline had been warning for years that this exact system would fail in exactly this way, and they were not heeded, and I want to write in their defence, because I have spent my career among their equivalents, the people who keep the lights on and are heard only in the hour after the lights go out.

There is a concept that the software trade uses and the rest of the world would do well to borrow: technical debt. When you build a system quickly, or cheaply, or leave an aging one unrenewed because renewal is expensive and the thing still runs, you take on a debt, and like a financial debt it accrues interest, silently, in the growing fragility of the system, the mounting risk that it will fail under a load it once could have borne. You do not see the interest accumulating. The system keeps working, quarter after quarter, and every quarter you decline to pay down the debt looks, on the books, like a saving, a cost avoided, a budget met. The historian of technology David Edgerton has spent his career arguing that we systematically misunderstand technology because we are mesmerised by innovation, the shiny new thing, and blind to maintenance, the unglamorous work of keeping the existing thing running, even though maintenance is where almost all the real labour and nearly all the real risk actually live. Southwest's meltdown is Edgerton's thesis written in a million stranded passengers: an organisation that invested in the visible and new while deferring the invisible work of maintenance, until the deferred bill arrived, all at once, in a storm.

The counterargument is real and I have made decisions on its side myself, so I will not caricature it. You cannot stop everything to rebuild the foundations; the market rewards features, not refactors; a system that has worked for twenty years has, by working for twenty years, earned some presumption that it will keep working; and every hour and dollar spent replacing plumbing that functions is an hour and dollar not spent on the things customers actually see and pay for. This is true, and a business that gold-plated every internal system in paranoid anticipation of the hundred-year storm would be out-competed by a rival who shipped and grew. The discipline is not to eliminate technical debt, which is impossible and would be its own folly; it is to know how much you are carrying, to service it deliberately, and above all to distinguish the debt you can safely roll from the debt that sits under something that, if it fails, fails catastrophically and all at once. Crew scheduling for a major airline is the second kind, and the people who knew that said so, and the saving looked good on the books right up until the morning it cost a million passengers and a corporate reputation built over fifty years.

Here is what I have learned watching this pattern across every kind of organisation, and it is the part the meltdown stories always miss. The people who maintain the critical systems are structurally doomed to be ignored, because their success is invisible and their warnings are, by definition, about things that have not happened yet. When maintenance works, nothing occurs, no headline, no crisis, no gratitude, only a cost on the budget with no visible return, and the person defending that cost sounds like an obstacle to progress. When maintenance fails, the failure is spectacular and public, and only then, in the wreckage, is the quiet warner suddenly revealed to have been right all along, briefly, until the wreckage is cleared and the pressure to defer the next repair resumes. It is a thankless position, the keeper of the unseen foundation, and it attracts a particular kind of person, undramatic, conscientious, more interested in the thing working than in being seen to make it work, and organisations spend them freely because their virtue is precisely the kind that makes no noise.

So my sympathy this fortnight is entirely with the engineers inside that airline who watched this coming and could not make anyone with a budget believe it, because a warning about a future failure can never compete, in a quarterly meeting, with a certain present cost. They were not unlucky and they were not overwhelmed by weather. They were the people who keep the lights on, telling the people who sign the cheques that the wiring was failing, and being told, reasonably and profitably and disastrously, that the lights were still on so the wiring could wait. The lights are the last thing to go. That is exactly why, when they finally do, it is already too late to have listened.

The Ledger
AI
ChatGPT, one month old, has passed a million users and prompted a "code red" at Google. No enterprise product exists yet. Watch Microsoft.
Data centers & power
Not a front-page subject. The industry's story is still chips: Nvidia's H100 accelerator went on sale in the autumn, and nobody outside a research lab is short of power.
Rates
The US Federal Reserve raised its target range to 4.25 to 4.50 percent on December 14, its seventh increase of the year, and signalled more to come. The Bank of Japan's yield move on December 20 was the fortnight's real surprise.
Real estate
US 30-year mortgage rates ended the year at about 6.4 percent, roughly double where they began it. Transaction volumes have fallen sharply; prices, so far, only a little.
Healthcare
The US Covid public health emergency remains in force. China's exit wave is the largest uncontrolled outbreak in the world.
India tech
Startup funding fell by about a third in 2022 from the record set in 2021, and the layoffs that began mid-year continued into December. Services firms, so far, are still hiring.
What we called wrong
Nothing yet. This is the first issue.
The Back Page

Edson Arantes do Nascimento, 1940 to 2022

He said he scored 1,283 goals. FIFA counted fewer. It does not matter. He won the World Cup at 17 in Sweden, again in 1962, and a third time in Mexico in 1970 with a team people still name as the best there has been. He played his whole club career in Brazil, for Santos, when every European club that mattered wanted him, and then in New York when almost nobody in America knew what the game was. He was 82. The city of Santos will bury him on Tuesday, after a wake at the stadium.