ticoombs

joined 2 years ago
MODERATOR OF
[โ€“] ticoombs@reddthat.com 13 points 6 months ago

Thank you for the update and it's good to hear your upcoming plans. Being one of those people in Australia (Reddthat) it will be good to see if it actually works as it's designed too!
I'd love to save $7/m to not have a server dedicated to batching the federation traffic ๐Ÿ˜…

When you lay out the timelines for 0.19.3 onwards no time at all has gone by, and having to deal with the issues after .3 has certainly not been fun as an admin. (And I'm only a small server compared!)
Being such a huge player in our Lemmyverse, thanks for taking the time to plan this out as I know how much testing has been done to get us this far.

It's always a nice experience chatting to the LW team!
Hope your updates go smoothly!

[โ€“] ticoombs@reddthat.com 2 points 6 months ago

We got there in the end ๐Ÿ‘ and welcome aboard!

[โ€“] ticoombs@reddthat.com 2 points 6 months ago (2 children)

Oh wow... I miss interpreted your intention. I thought you meant promote as in, using that post as a way to promoting someone to join, not that you wanted to be promoted into the position!

Sorry for the mix-up and glad to have you on board. It's been actioned!

[โ€“] ticoombs@reddthat.com 2 points 6 months ago

Appointed ! Happy to have you here

[โ€“] ticoombs@reddthat.com 2 points 6 months ago (4 children)

Yeah saying I endorse the request to find moderator.

Keep in kind that all reports on Lemmy get sent to the admins of the reported user, and wherever the community is hosted. So even if you report a post or comment in woman's hockey, it will always be seen by Reddthat admins (& the moderator). So I and the other Reddthat admins already see every report (for our instance).

Also anyone on Lemmy can be a moderator of it, it doesn't just have to be the people on Reddthat, so you might want to post to the actual community asking if anyone wants to be a moderator.

I'll pin it to the community and hopefully you get some traction, or you yourself can be the mod for the interim and then you can pin it yourself :)

[โ€“] ticoombs@reddthat.com 3 points 6 months ago (6 children)
[โ€“] ticoombs@reddthat.com 2 points 7 months ago

Games do not fall under this law, as they are already have laws around age restrictions. So if you can buy Minecraft they you are allowed to play Minecraft..

[โ€“] ticoombs@reddthat.com 4 points 7 months ago

Verification only needs to be reasonable. For us a system something what you have suggested would be reasonable for us. And/Or use a content warning popup like other NSFW sites. By continuing you are agreeing that you are 16 years or older etc...

I think by doing that and modifying the signup process it will be reasonable enough. Considering these laws are clearly not targeted at us and any watchdog will be targeting the "major" social medias like that TokTok site and the FaceSpace.

I was initially super sceptical about it but I think there is a way to navigate it while not compromising our ideals. (Which I kinda need to write in a more concrete fashion!)

[โ€“] ticoombs@reddthat.com 2 points 7 months ago (2 children)

To be fair on the EFF it has been less than a week and the requirements are still very much in the air. The minimum user count is drumroll 2. I forgot to include this lovely picture for the exact wording.

[โ€“] ticoombs@reddthat.com 2 points 7 months ago

Thanks. I might wait until .8 then.

[โ€“] ticoombs@reddthat.com 2 points 7 months ago

I too would love to know what your experiencing (so I can fix it!)

[โ€“] ticoombs@reddthat.com 5 points 7 months ago (1 children)

Having a list of all alternatives in the sidebar of alternative places to find people is my recommendation. Thus if people wish to keep a backup community on Reddthat I'm sure to let them.

 

We had a brief outage today due to the server running out of space.

I have been tracking our usage but associated it with extra logging and the extra build caches/etc that we've being doing.

Turns out the problem was the frontend Next-UI which has been caching every image since the container was created! All 75GB of cached data!

Once diagnosed it was a simple solution to fix. I'm yet to notify the project of this error/oversight and I'll edit this once Issues/PRs are created.
I also haven't looked at turning the caching off yet as my priority was recovering the main Reddthat service.

Thanks all for being here!

Tiff

 
 
 

Recently I've taken the docker compose example from SChernykh and have started a p2pool for Reddthat!

https://github.com/SChernykh/p2pool/tree/master/docker-compose (many thanks here!). After some minor changes I removed the IP listing from statistics and increase the visibility to 100 "supporters". It's viewable at donate.reddthat.com. (if @admin@monero.town wants the code change I can provide diff)

The idea was to also allow people to donate to instances via CPU instead of actual $.

My question for the community is whether I am creating a centralised pool or am I still participating in a decentralized fashion?

 

here's a graph showing when we did the deploy!

43
submitted 11 months ago* (last edited 11 months ago) by ticoombs@reddthat.com to c/reddthat@reddthat.com
 

Hello. It is I, Tiff. I am not dead contrary to my lack of Reddthat updates ๐Ÿ˜… !

It's been a fun few months since our last update. We've been mainlining those beta releases, found critical performance issues before they made it into the wider Lemmyverse and helped the rest of the Lemmyverse update from Postgres 15 to 16 as part of the updates for Lemmy versions 0.19.4 and 0.19.5!

Thank-you to everyone who helped out in the matrix admin rooms as well as others who have made improvements which will allow us to streamline the setup for all future upgrades.

And a huge thank you to everyone who has stuck around as a Reddthat user too! Without you all this little corner of the world wouldn't have been possible. I havn't been as active as I should be for Reddthat, moderating, diagnosing issues and helping other admins has been taking the majority of my Reddthat allocated time. Creating these "monthly" updates should... be monthly at least! so I'll attempt start posting them monthly, even if nothing is really happening!

High CPU Usages / Long Load Times

Unfortunately you may have noticed some longer page load times with Reddthat, but we are not alone! These issues are with Lemmy as a whole! Since the 0.19.x releases many people have talked about Lemmy having an increase in CPU usage, and they have the monitoring to prove it too. On average there was a 20% increase and for those who have single user instance this was a significant increase. Especially when people were using a raspberry pi or some other small form factor device to run their instance.

This increase was one of the many reasons why our server migrations were required a couple months ago. There is good news believe it or not! We found the issue with the long page load times, and helped the developers find the solution! -

This change looks like it will be merged within the next couple days. Once we've done our own testing, we will backport the commit and start creating our own Lemmy 'version'. Any backporting will be met with scrutiny and I will only cherry-pick the bare minimum to ensure we never get into a situation where we are required to use the beta releases. Stability is one of my core beliefs and ensuring we all have a great time!

Donation Drive

We need some recurring donations!

We currently have $374.10 and our operating costs have slowly been creeping up over the course of the last few months. Especially with the current currency conversions. The current deficit is $74. Even with the amazing 12 current users we will run out of money in 5 months. That's January next year! We need another 15 users to donate $5/month and we'd be breaking even. That's 1 coffee a month.

If you are financially able please see the sidebar for donation options, go to our funding post , or go directly to our Open Collective and signup for recurring donations!

Our finances are viewable to all and you can see the latest expense here: https://opencollective.com/reddthat/expenses/213722

  • OVH Server (Main Server) - $119.42 AUD
  • Wasabi S3 (Image Hosting) - $16.85 AUD
  • Scaleway Server (LemmyWorld Proxy) - $6.62 AUD

Scaleway

Unfortunately until Lemmy optimises their activity sending we still need a proxy in EU, and I havn't found any server that is cheaper than โ‚ฌ3.99. If you know of something with 1GB RAM with an IPv4 thats less than that let me know. The good news is that Lemmy.ml is currently testing their new sending capabilities so it's possible that we will be able to eventually remove the server in the next year or so. The biggest cost in scaleway is actually the IPv4. The server itself is less than โ‚ฌ1.50 so if lemmy.world had IPv6 we could in theory save โ‚ฌ1.50/m. In saying all this, that saving per month is not a lot of money!

Wasabi

Wasabi S3 is also one of those interesting items where in theory it should only be USD$7, but in reality they are charging us closer to USD$11. They charge a premium for any storage that is deleted before 30 days, as they are meant to be an "archive" instead of a hot storage system.

This means that all images that are deleted before 30 days incur a cost. Over the last 30 days that has amounted to 305GB! So while we don't get charged for outbound traffic, we are still paying more than the USD$7 per month.

We've already tried setting the pictrs caching to auto-delete the thumbnails after 30 days rather than the default 7 days, but people still upload and delete files, and close our their accounts and delete everything. I expect this to happen and want people to be able to delete their content if they wish.

OVH Server

When I migrated the server in April we were having database issues. As such we purchased a server with more memory (ram) than the size of the database, which is the general idea when sizing a database. Memory: 32 GB. Unfortunately I was thinking on a purely technical level rather than what we could realistically afford and now we are paying the price. Literally. (I also forgot it was in USD not AUD :| )

Again, having the extra ram gives us the ability to host our other frontends, trial new features, and ensure we are going to be online incase there are other issues. Eventually we will also increase our Lemmy applications from 1 to 2 and this extra headroom will facilitate this.

Donate your CPU! (Trialing)

If you are unable to donate money that is okay too and here is another option! You can donate your CPU power instead to help us mine crypto coins, anonymously! This is a new initiative we are trialing. We have setup a P2Pool at: https://donate.reddthat.com. More information about joining the mining pool can be found there. The general idea is: download a mining program, enter in our pool details, start mining, when our pool finds a "block", we'll get paid.

I've been testing this myself as an option of the past month as a "side hustle" on some laptops. Over the past 30 days I managed to make $5. Which is not terrible if we can scale it out. If it doesn't takes off, that's fine too!

I understand some people will be hesitant for any of the many reasons that exist against crypto, but any option to help us pay our server bills and allow people to donate in an anonymous way is always a boon for me.

Conclusion

These Lemmy bugs have been causing a headache for me in diagnosing and finding solutions. With the upcoming 0.19.6 release I hope that we can put this behind us and Reddthat will be better for it.

Again, thank you all for sticking around in our times of instabilities, posting, commenting and engaging in thoughtful communications.

You are all amazing.

Cheers,

Tiff

 

the "official" incident root cause analysis

 

An article from July, but I bet you haven't updated your bios! Or you left it open on purpose?

 

A really good article on how TPMs work

view more: โ€น prev next โ€บ