We just updated Lemmy to 0.19.12 which should have finally fixed the issue with accumulating browser cache files slowing down the main frontend UI, and also switched to the latest beta.7 for the alternative Photon frontend.
Meta (slrpnk.net)
Here we can discuss anything about this Lemmy instance/server itself.
Our XMPP support chat: Movim or XMPP client.
Are you still in the works to move to Piefed?
It is still planned for later this year, but there are some potential blockers on the Piefed side with password hashing and now it seems less urgent to be honest.
For what it's worth, I don't like change, whether it's in the climate or in the software that I use.
Yeah I just discovered slrpnk is back online, how wonderful! Thanks for restoring it so quickly!
Welcome back Schmorp! ^^
So glad the instance is back up and running. Was a pleasant surprise today.
It's a Pride miracle! Welcome back SLRPNK...? πππ
Yes! We're unexpectedly, but joyfully fully back up and running! π
poVoq will have a more detailed writeup on what happened at some point.
But it's so, so damn nice to be back! ^^
I will have some time on the weekend to write about how a friend of mine solved the issue and what the way forward is. But definitly some lessons learned and plans how to avoid this in the future.
Ok, sorry, didn't get around writing something on the weekend, and I will need to keep it short now as well.
The issue was primarily caused by a recent change in IP assignment by our ISP in addition to what looks like an very recent bug in our firewall software (IPfire) in combination with some odd errors in the fallbacks that I can't fully explain.
So basically our ISP is not assigning a completely fixed IP (it would cost 20 euro a month extra and we would need to switch to a business contract for it), but during the first 3 years of operation the IP they assigned only changed 3 times or so. Recently however they started to reassign a new IP more often, and annoyingly they assign a temporary IP first and a few weeks later apparently switch it again to a more permanent one in a different subnet.
We had this issue a few times in the last months, but the dynamic DNS of IPfire always caught it within a few minutes and thus is wasn't a major issue. But before I left on the work trip I updated the Firewall software, which caused the IP to switch to the temporarily assigned one, but again the dynDNS updated everthing within minutes, so I assumed everything was working fine.
However when the ISP randomly switched the subnet again, some still unkown bug caused the dynDNS to fail and the few failsaves I had in place to inform me about IP changes also didn't work, which led me to assume there was a hardware failure in the firewall and thus no way to fix it remotely.
But a few days after, I recieved an automated email from one of the services we host, which made it clear that outgoing connections were still working and thus we started to investigate how to find an alternative way to get the true IP of the server. Ultimatly my friend resorted to port-scan approximately 500k IPs in the subnet we knew the new IP should be at and we found 20 or so IPs that had ports open for a XMPP server and thus were potential candidates for our server. Luckily it was among them and thus we were able to manually update the DNS entries and restore service.
There are some lessons lerned from that, especially that the dynDNS of IPfire seems unreliable and I already have some plans to switch to another software for that, as well as add additional out of band notifications on IP change.
In addition we will try to find a cheap KVM to install on the main firewall that is connected on a seperate IP to be able to connect to it directly and reboot / troubleshoot it more easily even if the main connection is lost.
Last but not least we are experimenting with a Wireguard tunnel on a rented VPS which might allow more stable connection and the same VPS could be used to host some vital services like the XMPP server that thus would remain accessible even if the main server goes down (however since accounts are linked to the Lemmy database, this is a bit tricky and likely needs some partial database replication on the VPS or so, as otherwise there is no way to log in when the main server is down).
Most of these improvements will only happen once I have physical access to the server again end of July, but for now the service seems stable and hopefully we will not run into other issues until then.
I will once again offer my services as a seasoned CTO in the hopes that I can help you plan around some things ;)
Happy cake day!
Great work bringing it back up. Being unexpectedly down is kind of Solarpunk, and so is getting help from a friend ;)
it's so great that it's back up and running! I lurk more than anything but this is my lemmy home. muito obrigado.
Friends really do make the world go 'round.
Well done!
Glad to hear slrpnk.net is up again
I wonder what @poVoq@slrpnk.net did to get it back up again... π€
Ended up being a software and not hardware issue and a friend of mine helped a lot with finding a solution.
Nice to see the status lights back on green!
Yay! Thank you poVoq and friend!
Good to hear. :)
Rofl, yes!! π³οΈβπ I don't know how we're back up, but so glad to have my real home on lemmy up again. :D
Whee! γγγγ π
Happy Pride!
Be Crime. Do Gay.
Just be sure not to talk to them >.>
How I feel whenever I go over to my friend's house who has a smart speaker, though not nearly so wholesome and a lot more sardonically.
and plant some fruit trees while you're at it.
Is it possible to grow a really small citrus tree in a pot and get fruit from it? I live in an apartment
It will depend on the exact conditions (temperature, sunlight) and how well you manage soil fertility and watering, but yes, it should be possible. Owari satsuma and Fukushu kumquat come to mind as options. Get a large pot though.
Do you (or anyone else) know if we have a guide somewhere on apartment gardening? I'm really interested in growing some stuff to save money and y'know avoid eating imported fruits and vegetables if possible
I had no idea Duolingo was assisting ICE, what a shame, theres ** LanguageTransfer ** which is opensource if anyone is interested
always love any post by our admins
Waking up this morning to find slrpnk back up was a special treat. A big thank you to everyone who helped make it happen behind the scenes :)
Glad to see things are back, thank you for communicating during the downtime and running a good instances.
Glad to be back home! Thank you guys for the effort to bring it back and for keeping us informed, you're great!
Maybe I am pointing out the obvious but I want to share 2 experiences considering opsec and alternative software.
First, what helps me most to establish a healthier and safer relation to tech and to the online world is approaching communities offline. Connecting to a local hackerspace/hacklab and sharing experiences with people there is not only a lovely way to learn about alternatives but also to create the potential of sharing them with others. Building Infrastruktur together, organizing workshops, sharing tools, you name it. Establishing connections and contributing in such spaces even if you dont consider yourself a hacker can go a long way. check out hackerspaces.org to find something in your neighborhood.
Second, whenever I use an online tool, i am asking myself: is there any legitimeate reason why i need to do this online?And if i cant think of anything, i invest a couple of minutes to research how i can do the task offline. For example: I need to invert the colors of a black and white image. plenty of search results point to websites which can do this for you. "just upload your picture here". But you can do this locally as well with free (and open source) tools like gimp. examples like gimp or libreoffice are a great start. But how do I find alternatives that are open source for concrete issues that i want to solve? I often use alternativeto.org. I type in the name of the website that offers a service or the name of a commercial product and it usually provides plenty of open source options. By learning how to do things locally, I built a broad set of skills over time and i learn a lot about the free Software ecosystem. The example i gave is of course very simple, but you can take this approach to services that are less intuitive, for example: how to split or merge a pdf. the less you do online and the less you need to trust third parties to provide you a service, the easier it becomes to pay attention to opsec when you really need it in the online world.
thanks for the update! happy pride month and happy hacking! βΆ
Those are excellent pieces of advice, thank you for sharing them with us! :D
(missed making a βWe are currently clean on OPSEC.β joke)
Nice new communities. Just subbed.
I try and stick to FOSS whenever I can. I think that just about anyone could have an old PC running Linux even if they still need a windows box for gaming or whatever. Endof10 is a great resource for soon to be arbitrarily obsolete computers
Oh, welcome back! I am eagerly awaiting the posts downtime report!
If you didn't yet, you should probably update the post you made on that other lemmy server, with the information that slrpnk.net is up again!
I edited the post on lemmy.world, otherwise I think there was no official post.
...That Markdown table really rendered nicely.