The Global IT outage yesterday.

[Pappy13]

I've seen quite a bit of misinformation being thrown around about the recent global IT outage that I'm sure most of you have heard of by now and working within the airline industry which was hit pretty hard by the outage for the last 15 years gives me a bit of an insight into the issue so I'd like to share some information.

The problem was apparently caused by an update to a piece of software called the Falcon Sensor that is developed by a company called Cloudstrike. It's software that protects PC's from malicious attacks. It's one of the most widely used pieces of software for PC's for large companies which is why so many businesses were impacted including many in the airline industry. Most individuals do not use Cloudstrike, so most individuals PC's were not impacted. The problem was a result of an update to the software that required a reboot of the PC and then got stuck in the reboot process preventing the PC's from booting, effectively completely disabling them.

As far as I'm aware American Airlines PC's had mostly recovered from the boot issues by 8:00 AM central yesterday. They were still having some issues with some applications and flights are probably still impacted, but the PC's themselves had recovered from the boot issue that plagued PC's running the Crowdstrike software. It might be a few days for all the applications to fully recover from the issue and for the airlines to get back on schedule, but jets were only grounded for a short time for their own safety because the FAA mandates that you can't fly aircraft without being able to track them and many of the PC's that are used to track the flights were disabled.

I saw an article that mentioned that Southwest Airlines was not impacted because they were running on Windows 3.1 which is not correct. The reason that Southwest Airlines was not impacted is simply because they do not use the Crowdstrike software, they use a different piece of software for their PC security. I can assure you that Southwest Airlines is running Windows 10 on their PC's just like almost all large businesses are because upgrading to Windows 11 is a huge undertaking as all applications have to ensure they work with Windows 11 before it can be rolled out company wide and that takes a lot of time. American Airlines is also still running Windows 10 and expects to upgrade to Windows 11 sometime next year.

I'd like to point out that this is not a Microsoft outage as I've seen widely reported. My understanding is that the only PC's impacted were running the Cloudstrike software on PC's running Windows operating system. Cloudstrike actually has similar software that runs on other operating systems other than Windows, but each version is different and not all versions were impacted, only the Windows version was impacted. It's still a little unclear whether the reboot issue caused by Cloudstrike caused other issues within Windows and the cloud for example, but the fact remains that the source of the problem was the Cloudstrike software, not Microsoft software. You wouldn't call this an American Airlines outage because American Airlines was running Cloudstrike on Windows, so it's baseless to call this a Microsoft outage because it's operating system was impacted by that same Cloudstrike software. At least that's my understanding at this time, I'm sure that more information will come out later and perhaps Microsoft is more to blame than I currently understand, but I haven't seen anything to suggest that yet.

Finally I saw another article that mentioned that Apple doesn't have these sorts of issues. While there is some truth to that for reasons that I won't get into here the main reason is probably not what you think. Apple computers make up around 10% of the computers worldwide. Even if 100% of the Apple computers around the world were impacted by something like what happened yesterday to Windows PC's it wouldn't have nearly the impact simply because most large companies aren't using Apple computers. Apple computers can and do have issues from time to time, but comparing Apple to Microsoft is a bit like comparing apples to oranges (pun intended).

Anyway I hope that clears up a bit of the misinformation I've seen.

[Fau Teixeira]

Cloudstrike was the problem, it could have been prevented by a good software testing schema. The reason most larger companies were impacted is because cloudstrike is the higher tier of this type of software. Southwest not using cloudstrike means they went with a lower cost alternative.

falcon also exists for mac and linux but this update only impacted windows, so you are right in that it's not a windows issue.

[Spider-Dan]

I saw an article that mentioned that Southwest Airlines was not impacted because they were running on Windows 3.1 which is not correct.

Did you see this on The Onion?  

No one would be running Windows 3.1 in any sort of wide-scale deployment today.  That's not to say that there aren't companies running old software, but if they are, it would be far more likely to be MS-DOS (or something else older than Windows 3.1).  There really isn't any reason to stay on that particular version of Windows for 40 years.

Are there large companies still running Windows 95 somewhere today?  Probably.
Windows XP?  Definitely.
Windows 3.1?  No chance.

[Pappy13]

Did you see this on The Onion?  

No, the original article that I saw was actually legit or at least it came off as legit, but then it came out later that the source for that article was being sarcastic.

Just FYI, I saw today that the actual number of PC's affected was around 8.5 million or less than 1% of PC's running Windows. That's actually much lower than I expected. I think a bigger deal is being made of this issue then what was really there. I saw an article early on Friday that said it seemed like every single PC with Windows was crashing which I knew was overblown, but I had no idea how much that was overblown. A few very high profile companies were hit hard, but the vast majority of Windows PC's were not impacted. From the articles I read, I assumed it was actually much worse than it turned out to be.

[Pappy13]

I found another article about Southwest Airlines running Windows 3.1. You be the judge. Is this a legitimate article or are they making a joke? The odd thing is that the Crowdstrike outage affected around 8.5 million PC's, but that's less than 1% of all PC's running Windows. I guess they are assuming that over 99% of the PC's running windows are using Windows 3.1 or '95, that's a lot more than just Southwest Airlines.

https://www.msn.com/en-us/money/other/southwest-airlines-avoided-crowdstrike-microsoft-outage-because-it-s-still-running-windows-3-1-fourth-largest-us-airline-remained-free-of-bsod-errors-because-its-os-hasn-t-been-updated-in-decades/ar-BB1qJu8e?ocid=msedgntp&pc=ENTPHB&cvid=c65c5dfaa72f449d9bfded89d020f27d&ei=162

[Pappy13]

By the way, I have more information about American Airlines. The biggest issue that was encountered once it was clear what caused the problem was that AA also uses a software tool called bitlocker than prevents anyone from changing the configuration of a PC without the proper access. Every PC has an individual bitlocker key that must be entered if you try to reboot the machine using safe mode. Rebooting the PC using safe mode and removing the bad file from Crowdstrike was the only way to recover. It actually took much longer to find the necessary bitlocker code for every PC so that the bad file could be removed than it would have otherwise taken to recover. Still the issue was resolved around 7:00 AM on Friday when the problem first occurred around 1:00 AM.

Microsoft is now insisting that 3rd party software makers should not have access to the Kernal like Crowdstrike did so these issues can be avoided in the future. They apparently have said as much earlier but there was resistance to that idea because it was felt that 3rd party software vendors needed to have equal access to the Kernal to ensure equal access to security tools such as Crowdstrike (Microsoft provides it's own security tools). I think Microsoft might just persuade some people that it's actually better if Crowdstrike and other software makers don't have equal access to the operating system that Microsoft does.

Microsoft looks to revamp Windows access after Crowdstrike outage

[Spider-Dan]

Bitlocker isn't really a "software tool"; it's a built-in feature of Windows that encrypts the drive.  Windows automatically decrypts the drive on boot, but if you boot into Safe Mode then you need a keycode to decrypt the drive manually.  And yeah, getting those keys is a pain... you have to log in to a network system that stores the keys for every PC in the organization (but obviously, you can't use any PC that is crashed to get these keys).

Seems to me that MS might run into some more anti-trust issues if they try to deny kernel-level access to third-party vendors.  It may not even be legal for them to do this in the EU.

[Pappy13]

Bitlocker isn't really a "software tool"; it's a built-in feature of Windows that encrypts the drive.

Ah, I thought Bitlocker was a 3rd party, didn't realize it was a Microsoft product.

[Spider-Dan]

The current state of consumer protection law (HIPAA, etc.) is such that most organizations that run Windows will use Bitlocker to encrypt the drives of their PCs, which limits the scope of customer/patient data being exposed if a PC is stolen or lost.  It's really just the Crowdstrike program that was the problem.

[Pappy13]

I just read that the 8.5 million number of PC's impacted by the Crowdstrike issue was actually only the number of crash reports that Microsoft received. Obviously since you don't have to share a crash report, that number is lowballed. This was not smart by Microsoft, they either should have waited till they had a better estimate or made it clear that this was merely the number of crash reports received. See link below.

Microsoft admits 8.5 million Crowdstrike machines estimate was lowballed