this post was submitted on 31 Jul 2024
430 points (98.6% liked)

News

23296 readers
3179 users here now

Welcome to the News community!

Rules:

1. Be civil


Attack the argument, not the person. No racism/sexism/bigotry. Good faith argumentation only. This includes accusing another user of being a bot or paid actor. Trolling is uncivil and is grounds for removal and/or a community ban. Do not respond to rule-breaking content; report it and move on.


2. All posts should contain a source (url) that is as reliable and unbiased as possible and must only contain one link.


Obvious right or left wing sources will be removed at the mods discretion. We have an actively updated blocklist, which you can see here: https://lemmy.world/post/2246130 if you feel like any website is missing, contact the mods. Supporting links can be added in comments or posted seperately but not to the post body.


3. No bots, spam or self-promotion.


Only approved bots, which follow the guidelines for bots set by the instance, are allowed.


4. Post titles should be the same as the article used as source.


Posts which titles don’t match the source won’t be removed, but the autoMod will notify you, and if your title misrepresents the original article, the post will be deleted. If the site changed their headline, the bot might still contact you, just ignore it, we won’t delete your post.


5. Only recent news is allowed.


Posts must be news from the most recent 30 days.


6. All posts must be news articles.


No opinion pieces, Listicles, editorials or celebrity gossip is allowed. All posts will be judged on a case-by-case basis.


7. No duplicate posts.


If a source you used was already posted by someone else, the autoMod will leave a message. Please remove your post if the autoMod is correct. If the post that matches your post is very old, we refer you to rule 5.


8. Misinformation is prohibited.


Misinformation / propaganda is strictly prohibited. Any comment or post containing or linking to misinformation will be removed. If you feel that your post has been removed in error, credible sources must be provided.


9. No link shorteners.


The auto mod will contact you if a link shortener is detected, please delete your post if they are right.


10. Don't copy entire article in your post body


For copyright reasons, you are not allowed to copy an entire article into your post body. This is an instance wide rule, that is strictly enforced in this community.

founded 1 year ago
MODERATORS
 
  • Delta Air Lines CEO Ed Bastian said the massive IT outage earlier this month that stranded thousands of customers will cost it $500 million.
  • The airline canceled more than 4,000 flights in the wake of the outage, which was caused by a botched CrowdStrike software update and took thousands of Microsoft systems around the world offline.
  • Bastian, speaking from Paris, told CNBC’s “Squawk Box” on Wednesday that the carrier would seek damages from the disruptions, adding, “We have no choice.”
(page 2) 50 comments
sorted by: hot top controversial new old
[–] [email protected] 4 points 3 months ago

Revenge for the times they messed up my travel plans.

[–] [email protected] 10 points 3 months ago

Aw that’s a shame. Poor rich company.

[–] [email protected] 48 points 3 months ago (1 children)

Don't worry everyone... Each and everyone of the CEOs involved in this debacle will earn millions this year and next and will eventually retire with more money they could possible spend in 10 lifetimes

If anything, they'll continue to fall upwards completely deserving even more money

load more comments (1 replies)
[–] [email protected] 47 points 3 months ago (1 children)

No, POOR PLANNING and allowing an external entity the ability to take you down, that's what did it. Pretend you're pros, Delta, and be adequate.

Holy halfwit projection, batman.

[–] [email protected] 12 points 3 months ago (4 children)

The stories I could tell about how companies will hire a team to run tests on their digital and physical systems while also limiting access to outside nodes disconnected or screened from their core, primary, IMPORTANT systems.

Kicker is that plenty of people who work for these companies get it. Very rarely does someone in a position to do something about it actually understand. A few thousand dollars and they could have hired a hat or two to run penetration on systems and fixed the vulnerabilities, or at least shored them up so this fucking 000 bug didn't impact them so harshly.

But naaaaaaah. Gotta cut payroll, brb.

load more comments (4 replies)
[–] [email protected] 16 points 3 months ago

Good thing they got that $10 Uber Eats card.

[–] [email protected] 5 points 3 months ago
[–] [email protected] 19 points 3 months ago* (last edited 3 months ago) (2 children)

Sure, but they did send a $10 Uber Eats gift card, so you gotta take that into account.

[–] [email protected] 3 points 3 months ago* (last edited 3 months ago)

499,999,990. Could be worse.

Edit: oh I see someone else beat me to it

[–] [email protected] 3 points 3 months ago

Make it per affected device and then we can talk.

[–] [email protected] 64 points 3 months ago (6 children)

Why do news outlets keep calling it a Microsoft outage? It's only a crowdstrike issue right? Microsoft doesn't have anything to do with it?

[–] [email protected] 31 points 3 months ago* (last edited 3 months ago) (1 children)

The answer is simple: they have no idea what they are talking about. And that is true for almost every topic they are reporting about.

load more comments (1 replies)
[–] [email protected] 11 points 3 months ago (4 children)

Honestly, with how terrible Windows 11 has been degrading in the last 8 or 9 months, it's probably good to turn up the heat on MS even if it isn't completely deserved. They're pissing away their operating system goodwill so fast.

There have been some discussions on other Lemmy threads, the tl;dr is basically:

  • Microsoft has a driver certification process called WHQL.
  • This would have caught the CrowdStrike glitch before it ever went production, as the process goes through an extreme set of tests and validations.
  • AV companies get to circumvent this process, even though other driver vendors have to use it.
  • The part of CrowdStrike that broke Windows, however, likely wouldn't have been part of the WHQL certification anyways.
  • Some could argue software like this shouldn't be kernel drivers, maybe they should be treated like graphics drivers and shunted away from the kernel.
  • These tech companies are all running too fast and loose with software and it really needs to stop, but they're all too blinded by the cocaine dreams of AI to care.
[–] [email protected] 4 points 3 months ago* (last edited 3 months ago) (1 children)

They're pissing away their operating system goodwill so fast.

They pissed it away {checks DoJ v. Microsoft} 25 years ago.

load more comments (1 replies)
load more comments (3 replies)
[–] [email protected] 36 points 3 months ago* (last edited 3 months ago) (3 children)

It's sort of 90% of one and 10% of the other. Mostly the issue is a crowdstrike problem, but Microsoft really should have it so their their operating system doesn't continuously boot loop if a driver is failing. It should be able to detect that and shut down the affected driver. Of course equally the driver shouldn't be crashing just because it doesn't understand some code it's being fed.

Also there is an argument to be made that Microsoft should have pushed back more at allowing crowdstrike to effectively bypass their kernel testing policies. Since obviously that negates the whole point of the tests.

Of course both these issues also exist in Linux so it's not as if this is a Microsoft unique problem.

[–] [email protected] 6 points 3 months ago (1 children)

There's a good 20% of blame belonging to the penny pinchers choosing to allow third-party security updates without testing environments because the corporation is too cheap for proper infrastructure and disaster recovery architecture.

Like, imagine if there was a new airbag technology that promised to reduce car crashes. And so everyone stopped wearing seatbelts. And then those airbags caused every car on the road to crash at the same time.

Obviously, the airbags that caused all the crashes are the primary cause. And the car manufacturers that allowed airbags to crash their cars bear some responsibility. But then we should also remind everyone that seatbelts are important and we should all be wearing them. The people who did wear their seatbelts were probably fine.

Just because everyone is tightening IT budgets and buying licenses to panacea security services doesn't make it smart business.

load more comments (1 replies)
load more comments (2 replies)
[–] [email protected] 14 points 3 months ago (2 children)

It was a Crowdstrike-triggered issue that only affected Microsoft Windows machines. Crowdstrike on Linux didn't have issues and Windows without Crowdstrike didn't have issues. It's appropriate to refer to it as a Microsoft-Crowdstrike outage.

[–] [email protected] 28 points 3 months ago (2 children)

Funny enough, crowdstrike on Linux had a very similar issue a few months back.

[–] [email protected] 3 points 3 months ago (1 children)

It's similar. They did cause kernels to crash. But that's because they hit and uncovered a bug in the ebpf sandboxing in the kernel, which has since been fixed

[–] [email protected] 2 points 3 months ago (1 children)

Are they actually shipping kernel modules? Why is this needed to protect from whatever it is they supposedly protect from?

[–] [email protected] 2 points 3 months ago

They need a file io shim. That's gotta be a module or it'll be too slow.

[–] [email protected] 4 points 3 months ago (2 children)

I guess microsoft-crowdstrike is fair, since the OS doesn't have any kind of protection against a shitty antivirus destroying it.

I keep seeing articles that just say "Microsoft outage", even on major outlets like CNN.

[–] [email protected] -1 points 3 months ago (2 children)

To be clear, an operating system in an enterprise environment should have mechanisms to access and modify core system functions. Guard-railing anything that could cause an outage like this would make Microsoft a monopoly provider in any service category that requires this kind of access to work (antivirus, auditing, etc). That is arguably worse than incompetent IT departments hiring incompetent vendors to install malware across their fleets resulting in mass-downtime.

The key takeaway here isn't that Microsoft should change windows to prevent this, it's that Delta could have spent any number smaller than $500,000,000 on competent IT staffing and prevented this at a lower cost than letting it happen.

[–] [email protected] 2 points 3 months ago

The key takeaway here isn't that Microsoft should change windows to prevent this, it's that Delta could have spent any number smaller than $500,000,000 on competent IT staffing and prevented this at a lower cost than letting it happen.

Well said.

Sometimes we take out technical debt from the loanshark on the corner.

[–] [email protected] 3 points 3 months ago (5 children)

Delta could have spent any number smaller than $500,000,000 on competent IT staffing and prevented this at a lower cost than letting it happen.

I guarantee someone in their IT department raised the point of not just downloading updates. I can guarantee they advise to test them first because any borderline competent I.T professional knows this stuff. I can also guarantee they were ignored.

[–] [email protected] 5 points 3 months ago (1 children)

Also, part of the issue is that the update rolled out in a way that bypassed deployments having auto updates disabled.

You did not have the ability to disable this type of update or control how it rolled out.

https://www.crowdstrike.com/blog/falcon-content-update-preliminary-post-incident-report/

Their fix for the issue includes "slow rolling their updates", "monitoring the updates", "letting customers decide if they want to receive updates", and "telling customers about the updates".

Delta could have done everything by the book regarding staggered updates and testing before deployment and it wouldn't have made any difference at all. (They're an airline so they probably didn't but it wouldn't have helped if they had).

[–] [email protected] 0 points 3 months ago (8 children)

Delta could have done everything by the book

Except pretty much every paragraph in ISO27002.

That book?

Highlights include:

  • ops procedures and responsibilities
  • change management (ohh. That's a good one)
  • environmental segregation for safety (ie don't test in prod)
  • controls against malware
  • INSTALLATION OF SOFTWARE ON OPERATIONAL SYSTEMS
  • restrictions on software installation (ie don't have random fuckwits updating stuff)

..etc. like, it's all in there. And I get it's super-fetch to do the cool stuff that looks great on a resume, but maybe, just fucking maybe, we should be operating like we don't want to use that resume every 3 months.

External people controlling your software rollout by virtue of locking you into some cloud bullshit for security software, when everyone knows they don't give a shit about your apps security nor your SLA?

Glad Skippy's got a good looking resume.

load more comments (8 replies)
[–] [email protected] 2 points 3 months ago

Competent IT staffing includes IT management

load more comments (3 replies)
load more comments (1 replies)
load more comments (2 replies)
load more comments
view more: ‹ prev next ›