The whole world is now „enjoying“ the Black Friday of IT where independently of each other huge Microsoft Azure datacenter in US failed for several hours, and then popular EDR „Next-gen AV“ solution Crowdstrike Falcon deployed worldwide so fucked-up upgrade to their „sensors“ running on all the Windows workstations, laptops and servers, that they instantly went into BSOD and then bootloop. As the case still unfolding (check the Reddit megathread) there is still a lot of news we are going to hear, anyway for those affected until there is some more elegant solution the first known workaround is to physically log in to the station in safe mode and delete/rename a problematic .sys file(s) shipped in the screwed-up upgrade.
Other options as they came later:
- revert the station completely from VSC snapshot or Veeeam backup etc.)
- Some people claim that several (dozens) reboots if computer connected via Ethernet could manage to quickly get the patched version over net before BSOD happens. Definitely worth trying to cycle reboot machines until they finally boot.
- Group Policy script for machines in AD: https://gist.github.com/whichbuffer/7830c73711589dcf9e7a5217797ca617
Anyway this is how I did it the manual way before the other options were available:
The problematic file in our case is called C-00000291-00000000-00000010.sys and this is the SHA256 hash of the file you do not want to have on your stations and consider it as a IoC in the future someone is going to use it for some nasty DoS:
c532187b26eebf232179f76abd17a77a0d815215c352a6f45edb1ee02e9dba30 *C-00000291-00000000-00000010.sys
How I fix our stations:
TLDR: (1) have Bitlocker recovery keys ready, (2) Get somehow into Safe mode or mount the volume from WinRE or other system, (3) delete or rename Crowdstrike .sys file(s) causing BSOD.
- Find the Bitlocker keys on USB flash as BEK files, printed recovery keys in locked drawer or saved in AzureAD/Entra ID. (Good luck if you have encrypted drives and cannot find the Bitlocker keys anywhere! May the Force be with you.)
- When physically present at the problematic server or laptop or workstation, twice force poweroff (5 s button) when Windows is in the process of booting to force Automatic recovery blue screen where you click „Cancel“. In the next screen „Auto repair“ click „Advanced options“.
- Or, if the BSOD happened at the right time twice, it may go into another blue screen called „Recovery“ where you click „Show advanced recovery options“.
- Then other menus appear. Third option in the left column is „Fix issues“. Then „More options“, then „Startup options“ second item and Reboot. (Not sure what are exactly the English names, I had it localized to Czech when taking these photos.)
- After reboot, if the system drive is encrypted using BitLocker, you will need to press Enter and then either plug in USB flash with binary BEK Bitlocker keys in the root directory, or enter the numeric recovery key manually which you should have printed or saved to Microsoft AzureAD or your password manager. If you decrypt the drive, in the next blue screen press F4 or F5 to get to safe mode (F5 is with networking, but only Ethernet usually works, not WiFi.).
- Login as local or AzureAD admin previously logged in to that station. Or as regular user but you will neet do then escalate to admin privileges anyway by entering admin password. (Entering Windows Hello PIN or fingerprints may not work in safe mode.)
- Remove or rename the problematic files named C-00000291*.sys in C:\Windows\System32\drivers\CrowdStrike\ , and reboot. Problem fixed, Falcon sensor should no longer download this screwed up file Crowdstrike retracted it once the cause was discovered.
Below is a bat script I prepared using ChatGPT, and used on these stations while in safe mode to rename the problematic files and log it (wearing on portable USB SSD as going around the offices…) Windows partition must be on C:\ not other drive. (If loaded via WinRE or command line in the Blue screen Windows drive will NOT be in C: and the script needs to be modified accordingly.)
It takes me around 10 min for each station to fix it this way. Not very scalable if you have a corporation with thousands of laptops and servers and your entire infrastructure including Windows Hyper-V hypervizors, Domain controllers etc. is down. (It gets close to 20 min if I cannot find the Bitlocker keys and search them frantically AzureAD console where fortunately I have them all.)
The script saves a simple log file of the hostname to the folder of the script on your USB SSD or flash drive. Not very fancy and no big details in there, time was more important factor than quality and the first working script generated by ChatGPT was the one I started using to get the company working ASAP. Make better scripts if you have the luxury of free time and while your colleagues are fixing the most important stations using manual methods.
Download via Gist on GitHub: https://gist.github.com/brozkeff/e9e33e4b9340d85d23c9bb325a055698
@echo off
setlocal enabledelayedexpansion
:: Script to unfuck BSOD loop on Windows stations affected by Crowdstrike Falcon sensor screwed up update on 2024-07-19
:: by renaming C-00000291*.sys files in C:\Windows\System32\drivers\CrowdStrike
:: Reddit megathread: https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_error_in_latest_crowdstrike_update/
:: Quick and dirty solution by Martin Brozkeff Malec (brozkeff.net) with the help of ChatGPT
:: Blog article:
:: https://www.brozkeff.net/2024/07/19/black-friday-2024-07-19-crowdstrike-falcon-edr-bsod-fuckup-workaround-batch-script-in-windows-safe-mode/
:: Get current date and time for the log file name
for /f "tokens=1-4 delims=/ " %%i in ('date /t') do set date=%%i-%%j-%%k
for /f "tokens=1-4 delims=: " %%i in ('time /t') do set time=%%i-%%j
set datetime=%date%_%time%
:: Get hostname
set hostname=%COMPUTERNAME%
:: Define the log file name
set logfilename=%~dp0log_%datetime%_%hostname%.txt
:: Get OS version
for /f "tokens=4-5 delims=[]. " %%i in ('ver') do set osversion=%%i %%j
:: Navigate to the CrowdStrike directory
cd /d C:\Windows\System32\drivers\CrowdStrike
:: Write debug info to log file
echo Hostname: %hostname% > "%logfilename%"
echo OS Version: %osversion% >> "%logfilename%"
echo >> "%logfilename%"
:: Rename the files and log the actions
echo Renaming files: >> "%logfilename%"
for %%f in (C-00000291*.sys) do (
set originalname=%%f
set newname=%%f.bad
ren "%%f" "%%f.bad"
if exist "%%f.bad" (
echo Renamed %%f to %%f.bad >> "%logfilename%"
) else (
echo Failed to rename %%f >> "%logfilename%"
)
)
:: Completion message
echo Operation completed. Log file created at: %logfilename%
endlocal
pause
Result of succesful run of the script. It just renames the files to *.old in the original folder, does not do anything else, does not delete them, you can use them later for forensic analysis or whatever.
Good luck fixing this IT nightmare in your company!
