this post was submitted on 19 Jul 2024
631 points (98.6% liked)

Technology

70919 readers
3327 users here now

This is a most excellent place for technology news and articles.

Our Rules

  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 2 years ago
MODERATORS
L3s@lemmy.world
enu@lemmy.world
technopagan@lemmy.world
L4s@lemmy.world
L3s@hackingne.ws
L4s@hackingne.ws
631
An angry admin shares the CrowdStrike outage experience (www.theregister.com)
submitted 10 months ago by lemmee_in@lemm.ee to c/technology@lemmy.world
136 comments fedilink hide all child comments
 

IT administrators are struggling to deal with the ongoing fallout from the faulty CrowdStrike update. One spoke to The Register to share what it is like at the coalface.

Speaking on condition of anonymity, the administrator, who is responsible for a fleet of devices, many of which are used within warehouses, told us: "It is very disturbing that a single AV update can take down more machines than a global denial of service attack. I know some businesses that have hundreds of machines down. For me, it was about 25 percent of our PCs and 10 percent of servers."

He isn't alone. An administrator on Reddit said 40 percent of servers were affected, along with 70 percent of client computers stuck in a bootloop, or approximately 1,000 endpoints.

Sadly, for our administrator, things are less than ideal.

Another Redditor posted: "They sent us a patch but it required we boot into safe mode.

"We can't boot into safe mode because our BitLocker keys are stored inside of a service that we can't login to because our AD is down.

(page 2) 50 comments
sorted by: hot top controversial new old
[–] Fedizen@lemmy.world 28 points 10 months ago

lmao

[–] phoenixz@lemmy.ca -3 points 10 months ago

Yet again: Switch to Linux.

[–] scottywh@lemmy.world -2 points 10 months ago (1 children)

If it only impacts a percentage of your machines then there was a problem in the deployment strategy or the solution wasn't worthwhile to begin with.

[–] phoenixz@lemmy.ca -2 points 10 months ago (2 children)

... So your point was that it would have been better if everything went down?

There are plentiful reasons why deployments are done in parts, and I'm guessing that after today strategies will change to apply updates in groups to avoid everything going down.

Also, dear God, stop using windows as a server, or even a client for that matter. If you're paying actual money to get this shit then the results are on you.

[–] scottywh@lemmy.world -3 points 10 months ago

Also, 😂

[–] scottywh@lemmy.world 0 points 10 months ago

No.

My main point was that crowdstrike has always been lazy man's garbage.

[–] Cossty@lemmy.world 70 points 10 months ago (9 children)

I didnt know so many servers still run windows.

[–] kent_eh@lemmy.ca 14 points 10 months ago (2 children)

My former employer had a bunch of windows servers providing remote desktops for us to access some proprietary (and often legacy) mission critical software.

Part of the security policy was that any machines in the possession of end users were assumed to be untrustworthy, so they kept the applications locked down on the servers.

load more comments (2 replies)
[–] Hotzilla@sopuli.xyz 32 points 10 months ago (2 children)

Issue is not just on servers, but endpoints also. Servers are something that you can relatively easily fix, because they are either virtualized or physically in same location.

But endpoints you might have thousand physical locations, and IT need to visit all of them (POS, info/commercial displays, IoT sensors etc.).

[–] Miaou@jlai.lu 2 points 10 months ago

Parent comment applies even more so to such endpoints imo

load more comments (1 replies)
[–] stoly@lemmy.world 8 points 10 months ago (1 children)

I can’t imagine how much work it would be to migrate all your services onto Linux. The problem was people adopting windows in the first place.

[–] douglasg14b@lemmy.world 10 points 10 months ago (4 children)

I love the Linux bros coming out of the woodwork on this one when this could have very well have been Linux on the receiving end of this shit show. Given that it's a kernal level software issue, and not necessarily an OS one.

It's largely infeasible to use Linux for many, most, of these endpoints. But facts are hard.

[–] save_the_humans@leminal.space 14 points 10 months ago* (last edited 10 months ago) (3 children)

Hey man, let us have this one. Any immutable/atomic distribution could have either prevented this or easily rolled back the update. Not to mention a Linux offering by something like Red Hat, for example, wouldnt recommend installing closed source third party kernel modules for exactly this reason. Not sure about the feasibility of these endpoints, but the way things are generally done on, and the philosophy of, Linux could very well have avoided this catastrophe.

load more comments (3 replies)
load more comments (2 replies)
[–] terminhell@lemmy.dbzer0.com 13 points 10 months ago

On prem AD. At least for my MSP's clients. Have been pushing hard last few years to migrate to azure.

load more comments (5 replies)
load more comments