Yeah every image will be stored on s3 so I think we will see even faster speeds. Hope so!
So far we have managed to not block any other instance and I hope that can continue. :) I don't feel instance owners should make decisions for their users. Besides, it's going to become possible to block instances on user level in Lemmy soon I think.
Thank you :) I just wish there was a way to fix this in the Lemmy software. Currently im thinking maybe to move all images to object storage in S3. That seems to be the way people are doing it to get cheap storage.
I'm very sorry about that - it's because i still haven't figured out a way to disable Lemmy caching all images, so the disk fills up and eventually I have to manually delete all images. And since Lemmy stores cached images in the same place as user avatars and banners, they also get deleted...
I have some more things to try though. I will post as soon as I've fixed it... Hope you guys can be patient for a bit longer. :)
They are also working on this feature in Lemmy itself but still in progress: https://github.com/LemmyNet/lemmy/pull/3897
Good work man, I hope you keep it up even when you dont feel like going. My kid is doing strength training and Im amazed how he never skips a workout. Im more like... meh, I can do it tomorrow... and I wont.
I noticed this also. Since all pictures are handled by pict rs, deleting anything from disk risks removing any picture users have uploaded also.
Interesting design choice of Lemmy to keep both cached images and user images in the same place with the same names... They all look the same on disk with generated names.
I noticed today that community pics are gone also.
Try to upload the same images again and if that doesn't work, try to create new images to replace them. I will also try tomorrow to look into this more...
I have been reading up on pict-rs and it seems that the environment variable PICTRS__MEDIA__RETENTION__PROXY is the one to set for cleaning up old cached images.
It seems they added this environment variable in pict-rs version 0.4.2 so I bumped our pict-rs version from 0.4.0 to 0.4.2 and applied the environment variable now.
Hopefully this will take care of the disk space growing on its own, but will keep an eye on it. :)
I did stress a bit over it honestly but it was a quick fix at least. I have an idea now how to fix this situation for the future - there is an environment variable that can be set for the pict-rs software that will remove any cached image not viewed after X days. So I will try that one and see if it gets us disk back. :)
I feel the same. I use Lemmy a lot since it feels like the early days of the internet, but there is not enough interesting content to stick around for hours. Maybe it will grow but even if it doesn't, it's cosy and gives time for other things in life. :)
This is a really good and detailed post. I didn't see it when you posted but let me go through this... Seems really interesting and could be a huge Lemmy bug.
Yeah it was the first downtime we have had since we started! :)
I think it's an excellent idea to make it purge the oldest X days of cached images. I will look at your link and see what I can do. Thank you!
The instance itself is actually running in Hetzners server hall in Oregon, and it's been extreamly reliable. Not a single issue with them since we started this instance. :)
Now we will store images on s3, which means all image loadings will be from s3 to you users directly, and that means the instance doesn't even have to send you any images anymore - just the link to them on s3. So that will reduce load on the instance and help to make it even faster.
I'm very curious myself how this will be after the migration. Excited for it. :)