Apologies for the downtime.
We had some severe weather come through on Tuesday that caused extensive damage to power infrastructure throughout the area. I thought my location had been spared the worst of it, but a tree decided to fall on the transmission line servicing this area Wednesday morning (according to the power company, anyway). By that point, there were about 55,000 other outages ahead of my area in line awaiting repairs (no hard feelings there).
Since power outages are rare here, at least ones lasting longer than an hour or two, I only keep about 2 gallons of fuel on hand for the generator. At typical 3/4 load, that usually last about 4-5 hours. Again, long outages are extremely rare here, so that's usually more than enough runtime. Sadly, this outage lasted much longer (even though I was able to stretch the generator runtime by slightly hacking my UPS)
Federated content should now be coming back in, but it'll take a while to catch up.
Lessons learned:
- Mother nature is a badass bitch. I'm going to start keeping at least 2 days of fuel on hand. Worst case, I'll throw it in the car before it goes stale.
- My 48v e-bike battery and stepdown converter can run my UPS and bare-minimum servers for about 6 1/2 hours (probably more; it was only about half charged and had sat untouched on a shelf for over a year). That came in handy and was why we were able to stay online as long as we did.
- My primary network provider is pretty solid. That stayed up longer than I thought it would.
- Failover to the backup WAN works but failing back to primary does not. That's a "me" problem to fix in my watchdog script.
"Why don't you just throw it in the cloud like a sane person?" you may be asking. Well, it is and it isn't already. It's a hybrid setup. The UI and front end caches/proxies are all cloud based but the DB and API are located on prem where I can throw as much resources at them as I want for free.
I've been tempted to move those up, but it would cost more money than I want to devote to Lemmy at the moment (at least if I want to maintain the same level of performance).
Most of my VPS's are at capacity, but I am going to work on setting up a standby VPS that can scale up and keep the most recent backup dump there.
Mostly, I just don't want to have to rely on donations to keep DubVee online. Right now, all of its components are secondary payloads on my existing VPS hosts or are running on-prem on my own hardware (for free, for all intents and purposes).
I'd rather deal with an outage from time to time than have to constantly wonder if I'm going to be able to pay the cloud hosting bills. It's one of the reasons I've envisioned DubVee remaining relatively small.