CrowdStrike update previously crippled Linux systems, but went unnoticed

Debian systems crashed and refused to boot

In April, an update to CrowdStrike caused all Debian Linux servers at a civic tech lab to simultaneously crash and refuse to boot.

The update was not compatible with the latest stable version of Debian, despite this specific Linux configuration reportedly being supported.

After CrowdStrike acknowledged the issue a day later, it took weeks for them to come up with a root cause analysis.

The analysis revealed that the Debian Linux configuration was not included in their test matrix.

Insufficient testing and compatibility issues

Rocky Linux users reported similar issues after upgrading to RockyLinux 9.4, due to a kernel bug caused by a CrowdStrike update.

CrowdStrike support acknowledged the issue and noted a pattern of inadequate testing and lack of attention to cross-operating system compatibility issues.

The disruptions to Linux systems occurred without much awareness, raising serious concerns about CrowdStrike’s software update and testing procedures.

About the recent outage caused by the CrowdStrike update

The recent major outage of Windows computers around the world was caused by a “defect” in an update to CrowdStrike’s flagship security product, Falcon Sensor.

This defect caused Windows computers with Falcon installed to crash before they fully loaded.

“The issue has been identified, isolated, and a fix has been implemented,” CrowdStrike said in a statement.

The outages affected, among others, supermarket checkouts, airport departure boards, work-issued laptops/desktops, airport check-in systems, airline ticketing/scheduling platforms and healthcare networks.

Federal government responds to problems

The U.S. federal government has been notified of the CrowdStrike outage and is in contact with CrowdStrike and other affected agencies.

Several federal agencies were affected by the incident, including the Department of Education and the Social Security Administration.

Homeland Security said it is working with the U.S. cybersecurity agency CISA, CrowdStrike and Microsoft to “fully assess and address system outages.”

Patch and temporary solution for broken update

To resolve the issue, CrowdStrike has released a patch and a temporary workaround. This workaround can restore normal operation to affected systems until a permanent solution is found.

The company advised users to boot their computers into Safe Mode or Windows Recovery Environment, navigate to the CrowdStrike folder and delete the faulty “C-00000291_.sys” file.

However, this manual solution can be a major challenge for companies and organizations with a large number of computers/Windows servers in remote locations.