What's the ~~Cloud~~CrowdStrike thing going on? : outoftheloop

[–] [email protected] 4 points 3 months ago

Big time AV software used by a lot of conpanies issued an update that crashed all windows machines it was installed on. The only way to fix it is to boot into safe mode and then delete a file. On millions upon millions of servers and pcs. One by one.

[–] [email protected] 3 points 3 months ago

Including Mercedes formula1 team lol.

[–] [email protected] 12 points 3 months ago

**Clown Strike

[–] [email protected] 11 points 3 months ago (2 children)

To add to these guys, what it looked like in hospitals was all computers going blue screen of death on a loop. You would reboot and it would get to the desktop and go BSOD again. Communications with windows servers also went down.

The problem is, the safety plans are contingent on having windows working. There’s little to no contingent on no windows, people just expect it to always be there.

I and my team and all the individuals we ended up responsible for were fine by morning, but it was not a safe time.

911 was affected. Ambulance dispatch was affected. Many medical based institutions rely on windows and these security systems as well as airlines and such.

That said, please check on grandma, your favorite old uncle, single parent living solo, and all the rest today. There is no telling how many people are dead or injured in homes right now who couldn’t get through to emergency services last night. Maybe still can’t today.

[–] [email protected] 3 points 3 months ago

There is no telling how many people are dead or injured in homes right now who couldn’t get through to emergency services last night.

Waooooowww. I hadn't realized how harmful this issue was. It sounds like CloudStrike's future is bleak.

[–] [email protected] 16 points 3 months ago (1 children)

Intrinsic trust in outside actors in the kernel is a bad idea.

[–] [email protected] 15 points 3 months ago

State actors wish they could fuck shit up like Microsoft.

[+] [email protected] 64 points 3 months ago* (last edited 2 months ago) (4 children)

[deleted]

[–] [email protected] 2 points 3 months ago* (last edited 3 months ago)

I was wondering why this didn't happen to me, and I guess it's because my company uses Carbon Black which seems to be a competitor to CrowdStrike. Phew!

Thanks for the good explanation.

[–] [email protected] 4 points 3 months ago* (last edited 3 months ago) (1 children)

For context, I do not work in anything remotely close to an IT department. I work in a hospital. This affected my work the other day too. I am a bit more tech savvy than some of my coworkers, so I was attempting to see if I could fix the issue on my own by reverting to the previous windows update in recovery mode.

However, doing so prompted me for a Windows product key, which I obviously didn't have because I didn't install Windows on that computer.

The IT department had to come around individually for every single affected computer. They had to manually look up and type out the unique Windows product key for every single affected computer in order to be able to fix the problem.

Not sure if most installs of Windows act that way or not, but it definitely made the process more manual and annoying than it had to be. I have no idea why many of the recovery options required me to look up and enter a Windows product key. Seemed very odd to me and just made the ordeal more manual and time consuming than it had to be.

I believe some hospitals even ended up having to cancel surgeries.

[–] [email protected] 16 points 3 months ago (1 children)

Great response, and matches with what I've been reading.

Holy shit, someone(s) at CrowdStrike have had a reeeeally bad end of the week!

[–] [email protected] 5 points 3 months ago (1 children)

Agreed. It is odd that CrowdStrike doesn’t seem to?? Please correct me. That they don’t have a test release and a general release.

Weird

[–] [email protected] 5 points 3 months ago

It's definitely weird. They might well throw one out two developers and As under the bus for this event (I've seen this happen at other organizations - and such decisions smack of shitty management). But no matter how you look at it this is a company-wide failure. Because they didn't have the infrastructure and policies in place to test their changes properly.

[–] [email protected] 11 points 3 months ago

What a disaster! Thanks for the thorough response 😊

[+] [email protected] -6 points 3 months ago* (last edited 3 months ago) (1 children)

Edit: I got the name (Crowd Strike) and some basic facts wrong. There are other, better answers here.

[–] [email protected] 1 points 3 months ago (1 children)

I'm guessing the 33% vote rate is due to your intro, not your explanations?

[–] [email protected] 1 points 3 months ago

I got the name (Crowd Strike actually) and some basic facts wrong, and the intro probably didn't come off the way I meant it to.

[–] [email protected] 3 points 3 months ago

Crowd strike is an antivirus program that is installed on servers and laptops/desktops. The update corrupted a file that caused the operating system not to boot. The impact is thousands of hours of manual labor to recover these servers and endpoints. You have to do it in person unless your user is tech savvy enough to get into safe mode and delete a file. And you need admin rights

[+] [email protected] 0 points 3 months ago

[deleted]

Out of the loop