FIELD NOTE 002 · 23 JUL 2024

One Update, Millions of Machines: Systemic Risk Has a Face

One failed update at the centre, failures rippling out through first, second, third and nth-order rings.

On Friday afternoon, Australian time, check-in desks, supermarket tills, newsrooms and hospital systems went dark at almost the same moment. Nobody had been hacked. A single content update to a widely used security product had crashed the Windows machines it was meant to protect.

It is the clearest example in years of how one supplier’s change becomes everyone’s problem. The best way to read it is in orders of effect.

First order: the direct hit

CrowdStrike says a configuration update to its Falcon sensor, released at 04:09 UTC on 19 July, triggered a logic error that crashed Windows hosts. It was remediated 78 minutes later, but machines that had already taken it were stuck in a crash loop. Mac and Linux hosts were not affected.1 Microsoft estimates about 8.5 million Windows devices were hit, under one percent of all Windows machines.2

Second order: the services that stopped

Under one percent was enough, because those machines ran the services that matter. In Australia, the disruption hit media, retailers, banks and airlines.4 Recovery was largely manual: many affected machines had to be booted into safe mode or recovery and fixed one at a time.1 Remote fixes do not help a machine that cannot start.

Third order: the people outside the building

Customers who never heard of the supplier could not fly, pay or get through to the businesses they rely on. Even the message had to be managed: CrowdStrike and government agencies had to state publicly that this was not a cyberattack.1,5 For regulated entities, this is where contractual service levels, customer obligations and, from next July, APRA CPS 230 impact tolerances come into play.

Fourth order and beyond: the world after

Attackers moved within hours. CrowdStrike warned on the day of an eCrime actor distributing a malicious file posing as a hotfix,3 CISA reported phishing that exploited the outage,5 and the Australian Signals Directorate warned of malicious websites and unofficial code posing as fixes, while the Cyber Security Minister urged Australians to watch for scams and phishing.4 Beyond the week, expect the slower effects: insurers asking about concentration, questions over vendor contracts and liability, and regulators asking how many critical services rely on the same few suppliers.

Why it matters to the board

A point-in-time audit would likely have passed every organisation affected on Friday. The control was present and working, and it came from a reputable supplier. The risk was not in any single control. It was in the dependency: the same software, with the same deep access, running across most of the critical estate and updating everywhere at once.

Three questions for your next board meeting

  1. Which suppliers can push a change to most of our critical machines at once, and do we stage or test those changes?
  2. If our endpoints would not boot tomorrow, which business services stop, and how quickly could we restore them by hand?
  3. Who do we owe what in the first 24 hours (customers, regulators, counterparties), and is our communication ready before the scammers are?

How we help

Our systemic risk reviews trace exactly this chain, from first-order failure to fourth-order consequence, across your suppliers and business services. Red-team-forged consultants map where one change or compromise could travel, and tell the board which dependencies matter most. See Systemic risk review, or brief a CISO.

References

  1. CrowdStrike, “Technical Details: Falcon Update for Windows Hosts”, 20 July 2024. crowdstrike.com
  2. Microsoft, “Helping our customers through the CrowdStrike outage”, 20 July 2024. blogs.microsoft.com
  3. CrowdStrike, “Likely eCrime Actor Capitalizing on Falcon Sensor Issues”, 19 July 2024. crowdstrike.com
  4. Voice of America (Reuters), “Australia warns of ‘malicious websites’ after cyber outage”, 21 July 2024. voanews.com
  5. CISA, “Widespread IT Outage Due to CrowdStrike Update”, alert first published 19 July 2024. cisa.gov
← All field notes

Engage

Tell us what’s keeping your board up at night.