Hey folks!

We unfortunately had about half an hour of unplanned downtime today. This was caused by an issue with our hosting provider. The issue is solved for now, and I am planning to make some changes to prevent similar issues in the future. Sorry for the inconvenience!

Technical details

Our servers are communicating with our database over Hetzner's "vSwitch" service. Unfortunately, this service seems to be quite flaky - over the past few months, I have had to deal with the connection just dropping without recovering many times. Mostly this has not resulted in any noticeable downtime, as we have redundant servers, so even if one of them stops working, it won't affect lemm.ee users. However, in this instance, all of our API servers lost their connection to our database at the same time, which resulted in actual downtime.

I have now decided to migrate our setup away from the vSwitch in the near future to hopefully stop these issues for good. Should be possible to do this migration without any downtime, I just need to set aside some time to actually create an alternative solution for us, most likely over the coming weekend. I will update this post once the migration is complete.

Update: the migration is now complete! You can read more here.

all 12 comments

sorted by: hot top controversial new old

[-] Kroxx@lemm.ee 31 points 1 month ago

Thanks for the update, even with that little downage .ee is insanely stable!

[-] notabot@lemm.ee 21 points 1 month ago

Thanks for the update. If you have the time and energy after the migration I'd be interested to hear details of the new setup, we have servers using vswitch too, and while I haven't run into trouble myself I'd like to avoid it if possible. You know what they say about the wise learning from others' experience!

[-] Evil_Shrubbery@lemm.ee 19 points 1 month ago

In the future please schedule your unplanned downtime around my (also unplanned) pooping schedule, thank you. Today's experience was an almost barely noticeable inconvenience!

(Ofc I kid, lemm.ee is awesome!)

[-] Vent@lemm.ee 14 points 1 month ago

Thanks for your hard work!

[-] curled@lemm.ee 13 points 1 month ago

Thanks for the update and your hard work!

[-] aStonedSanta@lemm.ee 7 points 1 month ago

You rock!

[-] Legend@lemmy.sdf.org 3 points 1 month ago* (last edited 1 month ago)

Goddammit he's minerals how many times do i have to say this ?

[-] Zatore@lemm.ee 5 points 1 month ago

Thanks for telling us what happened! I know there's nothing we can do to help, but I'm always curious to know what happened. It can also be a great learning experience.

[-] casmael@lemm.ee 3 points 1 month ago

Thanks for the update - seems to be working for me now 🫡

[-] Legend@lemmy.sdf.org 3 points 1 month ago

Even with all these your instance seems to be one of the best ones out there lol nice work.

[-] gregorum@lemm.ee 2 points 1 month ago

FYI, image upload are broken again :(

this post was submitted on 06 May 2024

199 points (98.1% liked)

Meta (lemm.ee)

3473 readers

6 users here now

lemm.ee Meta

This is a community for discussion about this particular Lemmy instance.

News and updates about lemm.ee will be posted here, so if that's something that interests you, make sure to subscribe!

Rules:

Support requests belong in !support
Only posts about topics directly related to lemm.ee are allowed
If you don't have anything constructive to add, then do not post/comment here. Low effort memes, trolling, etc is not allowed.
If you are from another instance, you may participate in discussions, but remain respectful. Realize that your comments will inevitably be associated with your instance by many lemm.ee users.

If you're a Discord user, you can also join our Discord server: https://discord.gg/XM9nZwUn9K

Discord is only a back-up channel, !meta@lemm.ee will always be the main place for lemm.ee communications.

If you need help with anything, please post in !support instead.

founded 1 year ago

MODERATORS

sunaurus@lemm.ee