this post was submitted on 20 May 2025
143 points (98.0% liked)

Sysadmin

9222 readers
47 users here now

A community dedicated to the profession of IT Systems Administration

No generic Lemmy issue posts please! Posts about Lemmy belong in one of these communities:
[email protected]
[email protected]
[email protected]
[email protected]

founded 2 years ago
MODERATORS
top 28 comments
sorted by: hot top controversial new old
[–] [email protected] 11 points 1 day ago

Did any of the passengers on those 7,000 flights get any compensation? Or just a "so sorry, out of our control?"

Looks like delta tried to dismiss the passengers claims, but they are being sued for it.

[–] [email protected] 6 points 2 days ago

I work for a company that formerly used CrowdStrike. Since the event we no longer do.

[–] [email protected] 0 points 2 days ago* (last edited 2 days ago) (1 children)

When this happened it was pretty clear (to me, I was at least) that Delta didn't have an actual BCP. This is their CYA lawsuit and should be throw out - their negligence predates CloudStrike's incompetence.

[–] [email protected] 5 points 2 days ago

That notwithstanding, iirc the update was pushed to all of their "rings" even the n+2 or whatever.

Cs clearly fucked up and I don't see why they shouldn't be penalized for it.

[–] [email protected] 1 points 2 days ago

*clownstrike

[–] [email protected] 5 points 2 days ago

Its okay they gave ubereats vouchers. That should cover everything

[–] [email protected] 35 points 2 days ago (1 children)

Inb4 an out of court settlement happens in about a year and a half with CS not admitting fault but paying an undisclosed amount to Delta.

The Atlanta-based judge also let Delta pursue a computer trespass claim, and a narrowed claim that CrowdStrike fraudulently promised not to introduce an "unauthorized back door" into the carrier's computers.

Also, this will be interesting.

[–] [email protected] 20 points 2 days ago (3 children)

unauthorized back door

Isn't autoupdating software by definition an authorized backdoor by virtue of enabling it? The whole premise of CrowdStrike is continuous updates for attacks they see in the wild on other companies' systems.

Also if anything CrowdStrike did the opposite of a backdoor since everyone needed to find their BitLocker keys to get back in and clean this mess. It locked out the front and back door.

[–] [email protected] 2 points 1 day ago* (last edited 1 day ago)

Yes crowd strike is a huge security risk

[–] [email protected] 16 points 2 days ago (1 children)

There was an additional auto update function that wasn't disclosed. Delta had disabled the auto update because, like many large companies, they prefer to deploy changes incrementally so that an issue doesn't blow-up all their systems at once.

So...

Isn't autoupdating software by definition an authorized backdoor by virtue of enabling it?

Yes. Which is why they contend disabling it makes it unauthorized.

[–] [email protected] 1 points 2 days ago (1 children)

That's not how that works. CS didn't have at the time, an option to disable channel file updates. It's how their edr works. Delta's mssp or secops group, %100 knew this as it's in CS own documentation. They really don't have a foot to stand on here, but CS will pay it to make it go away.

[–] [email protected] 4 points 2 days ago (1 children)

CS didn't have at the time, an option to disable channel file updates

Yes, that's the crux of the accusation. Given the large number of people who seemed to be under the impression that selecting a staggered release cadence would protect them from a faulty update, it's not unreasonable to think that people were caught off guard by a second autoupdate system that they couldn't configure that could also tank their system.

[–] [email protected] 0 points 2 days ago

Before this, you could throttle the rollout for channel files. You could knock it down to 1 a minute if you wanted.

Channel files were not something that CS admins didn't know about.

[–] [email protected] 5 points 2 days ago (1 children)

I wouldn't call an auto update mechanism an unauthorised backdoor, it is required behaviour for that kind of software.

[–] [email protected] 6 points 2 days ago (1 children)

It's absolutely not required behavior! Software for servers has very different requirements from software for end users, and if you have a lot of them you also want to manage your end user machines differently.

Updates can go wrong, and if you roll out a bad update to everything at once you can crash everything and lose a lot of money. As aptly demonstrated by cloudstrike.

That's why Delta and many other companies disabled the auto update functions: so they could control the rollout cadence.
They reasonably believed that disabling autoupdates disabled them. They didn't expect a second autoupdate system that wasn't documented, wasn't controlled by the autoupdate system settings and couldn't be disabled.

[–] [email protected] 1 points 2 days ago (1 children)

It's not a second auto update. It's %100 documented in the software and you can %100 throttle it. Channel files are heavily discussed when you roll out CS.

[–] [email protected] 1 points 2 days ago* (last edited 2 days ago) (1 children)

https://www.crowdstrike.com/en-us/blog/falcon-content-update-preliminary-post-incident-report/

Might want to let crowdstrike know.

Rapid Response Content Deployment

Implement a staggered deployment strategy for Rapid Response Content in which updates are gradually deployed to larger portions of the sensor base, starting with a canary deployment.

Improve monitoring for both sensor and system performance, collecting feedback during Rapid Response Content deployment to guide a phased rollout.

Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed.

Provide content update details via release notes, which customers can subscribe to.

https://www.theregister.com/2024/07/23/crowdstrike_lessons_to_learn/

Maybe you're thinking of changes that they made as a result of the incident?

[–] [email protected] 1 points 2 days ago

No channel files where %100 there. It's in the general GUI settings. You could throttle channel files. Now after this your able to do General availability, Early availability or pausing them.

[–] [email protected] 15 points 2 days ago (3 children)

Can I sue them for the 13 people that threw shit at me because I couldn't get them their stupid addiction tickets?

[–] [email protected] 3 points 1 day ago (1 children)

... their stupid addiction tickets?

What are addiction tickets?

[–] [email protected] 2 points 1 day ago

Lottery tickets, usually scratch offs

[–] [email protected] 4 points 2 days ago

is Stupid Addiction a band name?

[–] [email protected] 7 points 2 days ago (1 children)

I feel sorry fot you but I must ask, was it literal shit?

[–] [email protected] 7 points 2 days ago (1 children)

No, just random items. Whatever was close. I got hit in the face with a can of pepsi. That wasn't fun.

[–] [email protected] 5 points 2 days ago (1 children)

Holy shit. No permanent damage?

[–] [email protected] 4 points 2 days ago (1 children)

Busted open my eyebrow but that was about it. I'm not trying to come off all tough guy but in comparison to other things I've gone through, it really wasn't a big deal.

[–] [email protected] 4 points 2 days ago (1 children)

I hope you're on my team when SHTF, then

[–] [email protected] 3 points 2 days ago

I'm on an island in the Atlantic Ocean that is not part of America and barely part of Canada. You got a trek to get over here and on my side lol