The CrowdStrike Incident and Windows Crash: What to do and How to fix it

A bug in a CrowdStrike software update instigated nothing less than a catastrophic IT outage on a global scale. The disruption was “not a cyberattack” according to a statement issued on X by CrowdStrike CEO George Kurtz. Instead, it was “a defect found in a single content update for Windows hosts” according to a statement from CrowdStrike.

Here’s a high level overview of how the “single content update” took down the Windows OS-based hosts that run critical IT for transportation, healthcare, manufacturing and other industries around the world:

1. CrowdStrike Falcon agents rely on regular software updates to combat malware on Windows hosts.
2. A software update from CrowdStrike, sent by CrowdStrike globally to all CrowdStrike Falcon agents, contained a bug.
3. The bug in the update from CrowdStrike caused Falcon agents to crash.

Fortunately, the engineers at CrowdStrike identified the bug and re-sent a clean update to all Falcon agents. However, many of the Windows hosts (both physical workstations and virtual machines) affected by the CrowdStrike Falcon bug were stuck in a loop of starting up and instantly crashing. This meant that many employees with CrowdStrike Falcon cybersecurity protection on their Windows PCs started their workday with a “blue screen of death” error.

How to Fix the CrowdStrike Blue Screen Problem

If your Windows PC with CrowdStrike Falcon is stuck at a blue screen of death, then follow these three steps to fix the issue:

1. Boot Windows into Safe Mode or the Windows Recovery Environment. Note that hosts encrypted using BitLocker will require a recovery key. If booting into Safe Mode, it is recommended to choose ” Safe Mode with Networking ” and use a wired network rather than WiFi.
2. Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory.
3. In that directory, delete any .sys files following the C-00000291*.sys naming pattern (e.g., C-00000291-1234.sys, C-00000291-12345.sys, et cetera).
4. After a normal reboot, your Windows PC will operate normally.

Blue Mantis recognizes that many IT departments do not configure Windows PCs with administrative privileges for employees. If that’s the case, then it could be difficult for remote employees to apply this fix without some kind of IT intervention. Yes, it could be resolved with a phone call between the affected employee and an IT professional, but obviously that option doesn’t scale very well. The rise of remote work coupled by lean IT departments means it’s likely these issues will impact operations at some companies for several days.

How to Fix the CrowdStrike Problem on a Windows VM

Resolving issues caused by the CrowdStrike update bug on a virtualized Windows host (either in the public cloud or in a virtual machine) can be accomplished in these steps:

1. Unmount the operating system disk volume from the impacted virtual server.
2. Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes.
3. Mount the volume to a new virtual server.
4. Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory.
5. In that folder, delete any files matching the C-00000291*.sys naming pattern (e.g., C-00000291-1234.sys, C-00000291-12345.sys, et cetera).
6. Unmount the volume from the new virtual server.
7. Mount the fixed volume to the impacted virtual server.

Why Did Microsoft Cloud Services Go Offline Due to CrowdStrike?

Microsoft was a victim of the CrowdStrike bug, because CrowdStrike is one of the solutions used by Microsoft to help secure their cloud infrastructure. Since the majority of Microsoft’s infrastructure for the Azure cloud, Office 365, and even their Xbox gaming services is built on the Windows OS, the bug in CrowdStrike’s software update caused Windows OS to crash. This is what created a cascade effect that temporarily took some Microsoft cloud services offline. Even though some reports claimed that Microsoft was responsible for the outages in Azure and Microsoft 365, the outages were due to the CrowdStrike bug applied to Microsoft’s Windows-based cloud infrastructure.

How to Avoid IT Infrastructure Downtime with Blue Mantis

Widespread IT service outages were frustrating for the millions of travelers who found themselves stranded in airports, the customers unable to complete transactions at their banks, or potentially life threatening for patients in hospitals impacted by the CrowdStrike software update bug. Regardless of whether or not the CrowdStrike incident was caused by a cyberattack, some of the methods used to mitigate the damage from ransomware and other attacks can be applied in this situation.

For example, redundant backups with high availability can help “roll back” software changes that adversely affect IT infrastructure. For companies invested in DevOps, it’s worth investing in a platform engineering solution with high security and automation in the development and integration pipeline. Most importantly, IT departments need to train their employees on how to respond to these crisis situations with repeatable processes and playbooks. Blue Mantis offers all these solutions—often as managed services that slot directly into your existing business processes.

Connect with us to assess your current environment and see how Blue Mantis can help fortify your IT infrastructure to avoid crippling outages and downtime.