IT outages are nothing new. This year alone, businesses in a range of sectors have experienced disruption, from the NHS[1] to McDonalds[2], from Sainsbury’s[3] to Three[4]. However, these were all relatively minor when compared to last month’s CrowdStrike outage.

When news of the issue first broke, you would have been forgiven for thinking cyber criminals were to blame. However, it soon came to light that the chaos stemmed from an update to CrowdStrike’s antivirus software, Falcon Sensor, which crashed devices running Microsoft software.  It soon came to light that the chaos stemmed from an internal issue with the testing process to updates issued to CrowdStrike’s antivirus software. Hackers are not the only threat to cybersecurity.

In the aftermath of this unprecedented outage, there are lessons for both businesses and software providers. While CrowdStrike has since committed to testing future updates more thoroughly and revising their testing processes, businesses should not be discouraged from enabling automatic updates on their devices. Software updates are often crucial to the security of a device.

What went wrong

Regarded as the most widespread IT outage in history, the CrowdStrike event brought businesses across the globe to a standstill. Thousands of flights were grounded, hospital operations were cancelled and people were left unable to access their bank accounts. For US Fortune 500 companies alone, the disruption is set to cost over $5bn[5].

In its own words, CrowdStrike, ‘released a content configuration update for the Windows sensor to gather telemetry on possible novel threat technique’[6]. In its review of the incident, CrowdStrike cited a faulty Content Validator that failed to detect a coding error in the update package[7], although stopped short of explaining why the Validator failed.

Over 8.5 million devices worldwide were affected by the outage[8], over 90% of which were back online within 24 hours. Naturally, older systems were harder to restore, and there are some that will never come back online. These will be forced to migrate to new systems. In the immediate aftermath of the attack, cyber criminals swung into action, looking to capitalise on weakened cyber defences through a wave of phishing attacks via email and SMS[9]. There is an element of irony that a cyber security company handed cyber criminals a golden opportunity to target its own customers.

Lessons for all involved

With such widespread disruption, the event highlighted the world’s dependency on Microsoft software. While nobody could have predicted the outage, all businesses must now assess their vulnerabilities, regardless of whether they were impacted by the CrowdStrike outage. Should a similar event happen again, businesses will be confronted with the age-old question. Should they turn to backups or try to fix the existing network? There is no one-size-fits-all approach. Ultimately, it depends on the depth of resource that a business retains. If an IT team are under equipped, backups will likely be the most reliable option, as manually fixing each desktop requires significant resources.

For cybersecurity firms and software providers, the CrowdStrike outage serves as an important reminder of best practice. While CrowdStrike have committed to running more robust checks on its Validator, all other providers should now review their practices to prevent this type of incident occurring again. Considering that the CrowdStrike outage impacted less than 1% of the devices that run Microsoft’s software[10], the impact of a future outage could be much more widespread. Given the impact that software updates can have on operating systems, the need to test updates in real world systems before pushing them out to the wider public is clear. Providers could also look to roll out updates in stages, by region for example, to prevent a global outage.

Despite the disruption caused by the CrowdStrike outage, it would be a mistake for businesses to disable automatic updates on devices. Previous automatic updates have provided a vital line of defence for businesses to prevent malware attacks and as cyber threats become ever more sophisticated, companies need to ensure they have every tool in their armoury.

[1] NHS Cornwall hospitals under pressure after IT outage - BBC News

[2] McDonald's blames global outage on third party - BBC News

[3] IT outage at UK supermarket Sainsbury's cancels "vast majority" of online orders – DCD

[4] Three apologises again as some still without mobile service - BBC News

[5] CrowdStrike outage explained: What caused it and what’s next (techtarget.com)

[6] Falcon Content Update Remediation and Guidance Hub | CrowdStrike

[7] CrowdStrike: 'Content Validator' bug let faulty update pass checks (bleepingcomputer.com)

[8] CrowdStrike outage explained: What caused it and what’s next (techtarget.com)

[9] Crowdstrike: Global cyber agencies warn about scammers - BBC News

[10] CrowdStrike outage explained: What caused it and what’s next (techtarget.com)