Technology

60016 readers

2664 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 2 years ago

MODERATORS

586

CrowdStrike broke Debian and Rocky Linux months ago, but no one noticed (www.neowin.net)

submitted 5 months ago by hal_5700X@sh.itjust.works to c/technology@lemmy.world

83 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] NutWrench@lemmy.world 11 points 5 months ago (5 children)

In April, a CrowdStrike update caused all Debian Linux servers in a civic tech lab to crash simultaneously and refuse to boot.

And then, you boot their servers from a Linux Live USB, run TimeShift to restore the last system snapshot, refuse the latest patch from Cloudstrike and they all lived happily ever after.

[–] avidamoeba@lemmy.ca 23 points 5 months ago

None of these things are used in actual server operations.

[–] Evilcoleslaw@lemmy.world 22 points 5 months ago

And it's not much more difficult to fix on Windows, except for the scale of the problem.

[–] RecluseRamble@lemmy.dbzer0.com 12 points 5 months ago (1 children)

Good luck doing that remotely. Which is the sole problem with this most recent CrowdStrike bug.

[–] friend_of_satan@lemmy.world 1 points 5 months ago

Anybody who doesn't already have ipmi serial console access set up needs to put that on their list of acceptance criteria for remediation of this incident.

[–] kurap1ka@lemmy.world 4 points 5 months ago

And on Windows you booted in safe mode and removed one file. What's the point of your post?

[–] friend_of_satan@lemmy.world 2 points 5 months ago

boot their servers from a Linux live usb

If I ran a computer lab that wasn't already net booted, I'd use this as the motivating factor to put that in place. Net booting to a repair image, or just reinstalling the whole OS either from scratch or a known good disk image, is where anybody who manages a fleet of computers should be.

There was a point in time where I had a pxe boot server vm set up on my laptop that I used to reload servers in our little row of racks at 365 main, because it let me quickly swap out the boot iso, and was faster than usb sticks were at the time.