I would like to roll back but the database schema changes would mean having to restore a backup. And potentially end up in more issues just like you were thinking too.
I guess it's best to wait for a fix, and I will also see if I can troubleshoot this myself a bit. I'm guessing it's a database issue since I can see very long running update statements on every restart, and they may not be able to complete for some reason.
I did another restart and your comment shows up in the thread.
So seems to be some bug that makes it stop federating after it has polled the queue once.
Indeed, it hasnt federated. So the restart of Lemmy polls the queue and then it stops working again. :/
Yeah I think the way outgoing federation works now in 0.19 has been rewritten to use a queue. So no comments or posts should ever be lost with this design, but there seems to be a bug causing the software to not process that queue sometimes I guess.
I think it was Lemmy software stopping to federate for some reason, and after i did a restart of Lemmy, it federated everything in the queue right away. But its worrying that this can happen, and I assume its a bug somewhere hiding in the software still.
I did a restart of Lemmy software just now, so I guess that made federation start to work again. But Im worried now it will stop again at any time. Will post a question in AskLemmy and see if anyone else has similar issues.
I can see the comment on kbin now after restart of Lemmy software:
It really seems like outgoing federation is buggy in this version. :/
I tried a quick restart of Lemmy to see if posts and comments are federated now after restart...
It seems to have made comments federate, but its worrying that it can stop working in this version. I have posted on AskLemmy, will see if anyone else has similar issues.
Thanks a lot, it made me look at this thread right away.
I posted above, but I think its still bugs in 0.19.1 that was supposed to be fixed in this version....
This seems very related to the update of Lemmy software to 0.19.1.
There was a critical bug with outgoing federation in 0.19 that was supposed to be fixed in 0.19.1:
https://join-lemmy.org/news/2023-12-20_-_Lemmy_Release_v0.19.1_-_Outgoing_Federation_fix
I would say it seems the bug is not fixed...
We changed some settings and now the alternate user interfaces are back. Keeping an eye on server performance... :)
Thank you, very good to know.
My idea was to try and see what specific query is failing in the database and go from there, so currently enabling logging of failed postgres queries. Hopefully see something in those logs...