this post was submitted on 27 Sep 2024
175 points (96.3% liked)

Ask Lemmy

26350 readers
1673 users here now

A Fediverse community for open-ended, thought provoking questions


Rules: (interactive)


1) Be nice and; have funDoxxing, trolling, sealioning, racism, and toxicity are not welcomed in AskLemmy. Remember what your mother said: if you can't say something nice, don't say anything at all. In addition, the site-wide Lemmy.world terms of service also apply here. Please familiarize yourself with them


2) All posts must end with a '?'This is sort of like Jeopardy. Please phrase all post titles in the form of a proper question ending with ?


3) No spamPlease do not flood the community with nonsense. Actual suspected spammers will be banned on site. No astroturfing.


4) NSFW is okay, within reasonJust remember to tag posts with either a content warning or a [NSFW] tag. Overtly sexual posts are not allowed, please direct them to either !asklemmyafterdark@lemmy.world or !asklemmynsfw@lemmynsfw.com. NSFW comments should be restricted to posts tagged [NSFW].


5) This is not a support community.
It is not a place for 'how do I?', type questions. If you have any questions regarding the site itself or would like to report a community, please direct them to Lemmy.world Support or email info@lemmy.world. For other questions check our partnered communities list, or use the search function.


Reminder: The terms of service apply here too.

Partnered Communities:

Tech Support

No Stupid Questions

You Should Know

Reddit

Jokes

Ask Ouija


Logo design credit goes to: tubbadu


founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[–] FigMcLargeHuge@sh.itjust.works 111 points 1 day ago* (last edited 1 day ago) (4 children)

A guy in our data center couldn't figure out who owned a particular machine that he needed to work on. So his solution to figure it out was to let them come to him. He went and pulled out the network cable and waited. He was escorted out a little while later. The moral of the story is don't go disabling production machines on purpose.

[–] Itdidnttrickledown@lemmy.world 1 points 7 hours ago

If you fear reprisal for a scream test then you need to make it look like an accident.

[–] Hugin@lemmy.world 44 points 1 day ago (1 children)

Where I worked we had a very important time sensitive project. The server had to do a lot of calculations on a terrain dataset that covered the entire planet.

The server had a huge amount of RAM and each calculation block took about a week. It could not be saved until the end of the calculation and only that server had the RAM to do the work. So if it went down we could lose almost a weeks work.

Project was due in 6 months and calculation time was estimated to be about 5 1/2 months. So we couldn't afford any interruptions.

We had bought a huge UPS meant for a whole server rack. For this one server. It could keep the server up for three days. That way even if wet lost power over the weekend it would keep going and we would have time to buy a generator.

One Friday afternoon the building losses power and I go check on the server room. Sure enough the big UPS with a sign saying only for project xyz has a bunch of other servers plugged into it.

I quickly unplug all but ours. I tell my boss and we go home at 5. Latter that day the power comes back on.

On Monday there are a ton of departments bitching that they came in an their servers were unplugged. Lots of people wanted me fired. My boss backed me and nothing happened but it was stressful.

[–] GreenKnight23@lemmy.world 31 points 1 day ago* (last edited 1 day ago)

I'd be super gluing those plastic toddler plug covers all over that thing.

fuck those other departments.

[–] superkret@feddit.org 39 points 1 day ago

Yeah, I've done that before – after asking literally everyone in IT, plus our external consultants, and getting the go-ahead from my team lead and the head of IT.

[–] ramble81@lemm.ee 88 points 1 day ago (2 children)

Honestly we do that when we ask and no one speaks up. Lovingly called the “scream test” as we wait to see who screams.

[–] slazer2au@lemmy.world 35 points 1 day ago (1 children)

Scream tests are a last resort though.

[–] intensely_human@lemm.ee 12 points 1 day ago (1 children)

Sounds like it was a last resort if he “couldn’t figure out” whose machine it was.

[–] superkret@feddit.org 11 points 1 day ago (2 children)

I don't understand how that is even possible.
Are there no logs? No documentation? Does everyone share an admin user with full rights?
I mean, there has to be a way to find out who accessed the machine last time.

[–] ramble81@lemm.ee 20 points 1 day ago* (last edited 1 day ago) (1 children)

You’d be surprised with inheriting tech debt. Quite often there’s no documentation, the last person to log in to the system is an admin that quit 3 years ago, but it doesn’t much matter because that’s only for a direct console login which normal users don’t do when accessing the application. With tribal knowledge gone and no documentation, only when you pull the network for a bit do you discover that there was this one random script running on it that was responsible for loading up all the needed data in the current system, when 9 of the other 10 times those scripts were no longer needed.

In a perfect world you’d have documentation, architecture and data flow diagrams for everything, but “ain’t nobody got time for that” and it doesn’t happen.

[–] superkret@feddit.org 6 points 1 day ago

Had that the other way around recently. A docker container failed to come back up after I had updated the host OS.
Was about ready to restore the snapshot, when I looked further back in the logs on a hunch.
Turns out that container hadn't worked before the update either. The software's developer is long gone, and no one could tell me what it was supposedly doing.

[–] GreenKnight23@lemmy.world 8 points 1 day ago

company a gets bought by company b. company b fires 50% of company a.

even a scream test won't get you answers because nobody is around that could complain nor know where the docs are.

[–] FigMcLargeHuge@sh.itjust.works 54 points 1 day ago (1 children)

I guess it depends on where you work. This was a large datacenter for a very large health insurance company. They made it a point later that day to remind people that it was a fireable offense to mess with production machines like that on purpose. And evidently the service he disabled was critical enough that it didn't take long for the hammer to come down. There were plenty of ways to find out who owned the machine, he just chose the easiest and got fired on the spot for it.

[–] intensely_human@lemm.ee 3 points 1 day ago (2 children)

So it wasn’t accurate when you said he “couldn’t” figure it out.

Well I am not him, so I can't tell you whether or not he actually "could" have figured it out. The options to figure it out did exist, but he chose not to use them giving it the appearance that he "couldn't". Are you this much fun at parties?

He couldn't figure it out, a competent person could have without unplugging it.