295
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 28 Jun 2023
295 points (99.7% liked)
Asklemmy
42525 readers
1030 users here now
A loosely moderated place to ask open-ended questions
Search asklemmy ๐
If your post meets the following criteria, it's welcome here!
- Open-ended question
- Not offensive: at this point, we do not have the bandwidth to moderate overtly political discussions. Assume best intent and be excellent to each other.
- Not regarding using or support for Lemmy: context, see the list of support communities and tools for finding communities below
- Not ad nauseam inducing: please make sure it is a question that would be new to most members
- An actual topic of discussion
Looking for support?
Looking for a community?
- Lemmyverse: community search
- sub.rehab: maps old subreddits to fediverse options, marks official as such
- !lemmy411@lemmy.ca: a community for finding communities
~Icon~ ~by~ ~@Double_A@discuss.tchncs.de~
founded 5 years ago
MODERATORS
Ye if you're adding like super redundancies etc sure. But I mean, it's lemmy. We don't need all that. My whole VM config is in ansible. I can literally scrap the whole thing, and redeploy it in 10 minutes. I just need to have a DB backup in case of some sort of catastrophic failure
IaC is awesome ๐ lemm.ee is deployed using Terraform.
My
lemmy_serverredundancy is mostly just so I can do infra changes without downtime (just take one node out of the load balancer, redeploy it and put the new one in), but it does also help a bit with general performance. I agree it's not strictly necessary, the vast majority of other instances are just running on a single server, but I do think it's very nice to have.For back-ups, I have point-in-time recovery, so I can restore the database to any random timestamp - and it has actually come in useful once already when two weeks ago I was able to restore to a good state about a few minutes after a problem - I think nobody even noticed that anything happened in that case ๐
Problem with Terraform is it assumes cloud providers, which tend to be expensive (or at least, I don't know a way to do terraform on hetzner dedis :D )
My solution is to get cheapskate VPS and dedis and loadbalance them as frontends. The VM request is manual, but I only have to do this once anyway. It's what I'm currently doing with the AI Horde. Of course, that doesn't help when there's DB changes but still.
What object storage are you using btw? I'm thinking to move to R2 or smt since I've had good experience with them until now. Contabo's is way cheaper but when I tried to use it for high-demand stuff it dropped dead on the spot. But it might be ok for Lemmy.
I can't thank you enough for sharing your knowledge, I am very interested in learning about server management and being able to read your thoughts is something I find extremely invaluable, please keep up with the great work! ๐
Yeah pretty much my setup. Full DB and pictrs backups uploaded to Backblaze B2 every few hours. Config files all backed up. Also send a copy to my home lab. Encrypted on the server before it gets sent out.
Worst case - we lose a few hours of data.
I keep about a week's worth of backups on B2 (and 2 days on my home server) which is just over 200GB now. But that costs next to nothing on B2.
Maybe eventually if my instance grows more I might consider doing a replica DB on another server.
My total costs are like, $7.50 a month but I only have 15 actually active users. I don't need to grow, but I'm willing to. If the costs increase too much I'll ask for donations and if that stops covering things I'll just close registrations. That's the great part of not being a business chasing infinite growth.