this post was submitted on 11 Oct 2023
25 points (100.0% liked)

Announcements

177 readers
2 users here now

Announcements/Changelogs go here.

founded 1 year ago
MODERATORS
 

Do not ever push to production without testing things first. I went and moved us to the beta branch because this commit caught my eye and I wanted us to have the emoji picker fixed: https://github.com/LemmyNet/lemmy-ui/commit/ae4c37ed4450b194719859413983d6ec651e9609

The beta branch was on dockerhub so I thought that it was at least minimally tested. I was sorely mistaken. Lemmy (the backend) itself would load, but lemmy-ui (the actual website that renders everything) would keep crashing when attempting to load. I couldn't roll back because the database was migrated to this beta version and it couldn't migrate back to the old version when attempting to launch said old version.

I had no choice but to restore from backup. We've lost a whole day's worth of posts (anything after 3AM CST.) I'm really really sorry.. blobcatsadpleading

I was just so excited to be able to unveil this, I didn't take my time with actually testing it.

you are viewing a single comment's thread
view the rest of the comments
[–] CookieJarObserver@burggit.moe 7 points 11 months ago (1 children)

Shit happens. One day is better than eight.

[–] Burger@burggit.moe 4 points 11 months ago (1 children)

Right. I thought I was restoring the most recent backup, but it turned out that the order for the archive list was in ascending order. I picked the top one thinking it was the latest. So that's why you saw 8 days when Burggit was back up for a split second.

My server automatically takes a backup once daily in the early AM, and keeps (I think? I'd need to check) 7 on hand. Any older than that, and they're pruned.

[–] CookieJarObserver@burggit.moe 4 points 11 months ago (1 children)

You should pick one day a week that is stored longer, if there is a repetitive issue it might corrupt over more than a few days.

Man i thought i did something wrong when the posts from the last week where gone

[–] Burger@burggit.moe 4 points 11 months ago (1 children)

Added a 1 per month backup retention. Thanks for the suggestion.

[–] CookieJarObserver@burggit.moe 3 points 11 months ago

Yeah had a problem once where a backup (made one once a day and keep 14) was overwritten by a corrupted data set and i basically lost most data, would be sad if that happens here if its avoidable👍