Announcements

242 readers

1 users here now

Announcements about system updates or other things related to this instance.

founded 1 year ago

MODERATORS

mrmanager

Lemmy.today was down just now (self.announcements)

submitted 1 year ago by mrmanager to c/announcements

5 comments fedilink hide all child comments

We just had another unscheduled downtime due to a linux kernel bug.

Yesterday we noticed some issues with the server - mainly that we couldnt stop some docker containers. As you know, we are running some extra user web interfaces for Lemmy and we noticed that they started acting weirdly and had one cpu running at 100% constantly.

I wanted to restart those containers but I couldnt stop them. Found some posts online that this is a bug in the Ubuntu linux kernel: https://github.com/moby/moby/issues/43094.

Our stop attempts caused the docker platform to behave weirdly and it started to affect the main Lemmy software, so we did a reboot of the server and installed the latest updates.

We are very sorry for this unscheduled downtime. :/ Did you guys notice weirdness with Lemmy in the last 8 hours or so?

top 5 comments

sorted by: hot top controversial new old

[–] megalow 8 points 1 year ago (1 children)

I didn't notice anything strange besides the downtime. Thanks for the update and information!

[–] mrmanager 4 points 1 year ago

Thank you :)

[–] 1984 4 points 1 year ago

Thanks for information, also didn't notice anything special really except a few minutes of downtime just now.

[–] Rivalarrival 3 points 1 year ago (1 children)

I've noticed some connectivity issues for the past couple days. Seemed like my client (Thunder) wasn't getting a response from lemmy.today, and eventually timed out. Most times I was able to sign out and back in immediately.

Shit happens, you're doing great.

[–] mrmanager 1 points 1 year ago

Yeah it's been a rocky ride since the update, and we had to restart the server a number of times to diagnose issues with outgoing federation that came with Lemmy 0.19.1.

Very sorry about that, and hopefully there is a fix for this coming soon from the developers. Thanks for being so nice about it. :)