We dont have anything restarting the server automatically, but you are on to something.
There have been occurances of the lemmy software and postgres restarting now and then since the update (several times per day). Postgres memory usage has been enormous, and we turned off the extra web user interfaces for a while to see if its still happening. So far its actually looking a lot better and memory usage looks stable, so maybe our hunch is right....
Yeah true, it's always something. :) I used to be on irc too back in the day, but sadly not anymore. But the worse big tech becomes, the more we will want to go back to other tech that didn't bother us and felt good to use.
OK thanks for the detailed information! I really hate to see that Lemmy still has federation issues. It seems like one of those core things that would just work by now. But any info and help about remaining problems is very valuable for the developers. I will read through what you said here and see if I can create a github issue from it.
No its the first time they specifically said there was a db update in the 0.19.0 upgrade and it took quite some time to complete too. 30 minutes or so of downtime.
Anyway, feels like we are finally past the problems... Things seems to work normally again now.
In theory yes, but it would have meant restoring a database backup and losing data that was created between the backup and the restoration.
We were considering it but it took a few days to realize it was a bug in Lemmy, and we always figured a hotfix would come quickly.
Unfortunately it was xmas and new years so it took a while for a fix to arrive. Next time we won't update the instance at such a bad time - lesson learned. :)
Gives me hope :)
This is the reaction I get from people too. :)
Yep here is the release notes also:
https://join-lemmy.org/news/2024-01-10_-_Lemmy_Release_v0.19.2_-_More_Federation_Fixes
Yeah we need the new 0.19.2 version of Lemmy to hopefully fix things. Good news is that I saw it being worked on yesterday on github, so it should come very soon now. :)
Yep I had to redo the fix also once more and also split up Lemmy in several docker containers. One container is doing the federation part only and I can restart that without disturbing the Lemmy user interface or database. So that one is restarting every 10 minutes now but we don't notice it since it's happening in the background.
Anyway, I read in the github thread that a fix for federation is working and will come soon... Just need to wait a bit longer.
I guess we just wait for the software fix. Something must be wrong there...
Yes you can use this one:
https://github.com/LemmyNet/lemmy-ansible