CS:doNOTgo – World Wide Broken Tech.

Well the last week in tech has been fun to say the least – BIG BAD Breaking News and disaster!

What Happened?

The title of this blog is a little tongue n cheek towards a fairy popular Game and at the same time reality for a POPULAR Cyber Security Firm’s Software. CrowdStrike (CS from here on out) came to a screeching holt when an update pushed out on the 18th July 2024 (NZ) – ticked off something in the windows OS causing MANY machines around the world to go BLUE. This may be one of the few times something like this has happened (iLoveYou virus and SQL worms being the other 2 that spring to mind – though those weren’t really weaponized & this one wasn’t either).

The Fix

There is a “fix” but if you couldn’t boot into windows you couldn’t do it, the fix was to rename one of the system files to something else or delete it entirely – essentially boot into safe mode navigate to Win System32 Crowdstrike drivers and delete the files by the name of C00000291-*.sys the system will reboot into working mode instead of BSoD / boot loop mode.
I personally suspect that Microsoft has not seen the number of reinstall in some time as that is the easiest way to “fix” this, although time consuming, especially if you have a large number of servers or workstations to work through.

In a blog post Crowdstrike issued the following statement:

CrowdStrike has corrected the logic error by updating the content in Channel File 291. No additional changes to Channel File 291 beyond the updated logic will be deployed. Falcon is still evaluating and protecting against the abuse of named pipes. 
This is not related to null bytes contained within Channel File 291 or any other Channel File. 

– Crowdstrike 20 July 2024, Executive Viewpoint
– Link: https://www.crowdstrike.com/blog/falcon-update-for-windows-hosts-technical-details/

The issue is fixed and the patch was issued all within about 6 or so hours from the time the world turned blue.

THE OUTFALL

if you are a relatively small company (~5 pcs) this may not be a huge issue for you but for those Bigger companies with 100s or thousands of pcs and windows servers this was a Pain In The Arse and a severe one at that, if your company has an I.T. team they were probably working on it from the time it came to the public notice and it took more or less all day if not 2 days to fully recover – the update was issued so any pcs that were running Crowdstrike’s Falcon software and came back online after the “failure” with a fresh install of windows and a new download of Falcon wouldn’t have seen any further issues this being said it is likely that people uninstalled Falcon or did a fresh install of windows and didn’t install Falcon and contacted Crowdstrike to End the account agreement in place, thus showing Crowdstrike what they thought. The Share market for Crowdstrike will likely be unstable, although I heard that some one had been “shorting the crap out of it days before” due to that it was already starting to drop as this took place but the real dive was just after the crash – Investors were like “stuff this mess we’re out!” causing a MASSIVE $23 Billion (yes Billion with a ‘B’) Crash, it is unlikely that the stocks will recover there is a chance – there is always a chance.

“No one will be able to pay for their Friday night beersies,” one person told the Herald.

source: NZ Herald
link: https://www.nzherald.co.nz/nz/crowdstrike-failure-kiwis-wake-after-night-of-chaos-following-global-it-outage/PKTVBS5TP5CFZN7SAEV3OWBIKM/

The pain was felt by 3 Banking institutions ANZ, ASB & KiwiBank, whose I.T. teams were probably working through the night to remedy the issue, with online payment down and POS & Till terminals playing up, it is lucky that the 4th Large institution (BNZ) and it’s subsidiaries (CoOperative Bank) had no issues paying for KFC – who also suffered some issues due to the Crowdstrike software update.
The widely used public charity ambulance service St. Johns also experienced issues, although they haven’t shared how it impacted them.
The NZ Government didn’t suffer any outfall from this update as “smiley” aka David Seymour Says:

“Earlier this evening I was briefed on three main issues relevant to New Zealanders affected by the global outage,” says Mr Seymour.

• Is it a malicious attack? No, it’s a glitch resulting from a software update from Crowdstrike.
• Is it critical? No. While it is very inconvenient to many people essential services are still working, systems and services are already coming back online as organizations apply the patch provided by CrowdStrike.  
• When will it be over? Organizations and businesses are moving quickly to apply the Crowdstrike patch, and systems are coming back online. Many have recovered their systems or are working to update them in the coming hours. While the fix is a straightforward one, it may take some organizations or systems more time to get back to normal. 

“The Government has worked quickly to understand the impacts of this issue and minimize them wherever possible.”
“We appreciate the inconvenience this is causing for the public, retailers and businesses. The Government is closely monitoring developments and will continue to provide updates.”

Source: Hon David Seymour (Acting Prime Minister, NZ)
link: https://www.beehive.govt.nz/release/update-global-it-outage

My Thoughts

I have waited until this point in this post to post my thoughts: I think its more intriguing if a Solar Storm took out one of the developers before they hit the “go” button on the update and that Developer was going to catch this error …
obviously this is leaning into the world of conspiracies etc… but I do think it’s more interesting.

But seriously now, I first found out about this when my Life Partner couldn’t log into her bank app or online banking and is typical in this house, googled the issue, checked “Down Detector” and found it was very wide-spread, at first she thought it was a Hack event (similar to that iLoveYou one), My first thought however was ‘that’s not a Hack, that’s something else’ and to be honest I couldn’t have been more right.
As it turns out it seams way too many organizations relied on one piece of software, akin to putting all one’s eggs in one basket. Though not all Windows PCs were affected and the 1% figure sounds really small there is 8Million or more windows PCs and if the math is done correctly 1% is 1 in every 100 … making therefore the number of affected PCs around 80,000 not insignificant, this is why you get headlines such as “the day the world blue screened”.

This isn’t the first time something like this has happened and likely won’t be the last, so until the next time we wait in the hope that it won’t be as bad.

All in all, if you search ‘crowdstrike’ on YouTube you will find many people showing you how to fix it or talking about it, a couple of these are:

Dave’s Garage (Dave Plumber [ex MS windows engineer])
https://youtu.be/wAzEJxOo1ts?si=VKggINL6rRPUX6zn

Network Admin Life
https://youtu.be/SH6_kkhwv5s?si=pRsQmScVLwxrtwS-

there are others that have weighed in on it but, in my opinion didn’t really have much to add apart from bringing it to light.

so until next time the Tech world (Almost) implodes
stay safe and be well.
Bryce