this post was submitted on 06 Oct 2023
21 points (100.0% liked)

Announcements

226 readers
1 users here now

Announcements about system updates or other things related to this instance.

founded 1 year ago
MODERATORS
 

Hi guys!

This weekend we will move lemmy.today over to using object storage for images. We will be serving images from Amazon S3 in the Oregon region (western USA).

The way lemmy software is designed right now, it caches every image federated from other instances. So even if we are small instance, we still have to store a lot of federated images locally on our disk. This leads to disk space running out quickly and we previously had to delete images because of this.

When we delete images, it removes not only those cached images but also user profile icons and banners, as well as community icons and banners. This is why we have some missing images under Communities right now, and also why users have lost their profile pics.

Its been very embarrassing to have to do this, and now we will move to object storage to prevent this from happening in the future. Its much cheaper compared to ordinary disk space and gives better performance for users, so its a win-win. We just need to do a one-time migration over to it.

**Estimated downtime hours: **

Oregon time: Sunday 3 am - 6 am

CET: Sunday 12 pm to 15 pm.

If you have any questions, you know what to do. :)


EDIT: Looks like it went well and images are now served from S3 instead of filling up our disks. :) The url to the images still looks like they are served by the instance, but thats by design appearently. In the background, they are fetched from S3.

Please fill free to re-upload any banners, user avatars or community pictures you had in place before that may have been broken by the disk cleaning before.

  • When you do, you have to create a new picture with a new name for Lemmy to actually replace the image. Otherwise it wont work - ive tried myself. :)

all 12 comments
sorted by: hot top controversial new old
[–] Buelldozer 5 points 11 months ago* (last edited 11 months ago) (1 children)

Out of curiosity what is this instance running on now? AWS with EBS and you're shifting storage to S3?

BTW Oregon region is my favorite, I have several instances based there for clients. Maybe not as many features as East but far better reliability in my experience.

[–] mrmanager 5 points 11 months ago

The instance itself is actually running in Hetzners server hall in Oregon, and it's been extreamly reliable. Not a single issue with them since we started this instance. :)

Now we will store images on s3, which means all image loadings will be from s3 to you users directly, and that means the instance doesn't even have to send you any images anymore - just the link to them on s3. So that will reduce load on the instance and help to make it even faster.

I'm very curious myself how this will be after the migration. Excited for it. :)

[–] Lewdiculous 2 points 11 months ago (1 children)

Very excited for it! Thanks for the improvements to infrastructure, manager!

[–] mrmanager 2 points 11 months ago

Hopefully all goes well. :)

[–] 1984 2 points 11 months ago
[–] Buelldozer 1 points 11 months ago (1 children)

Well done. Seems to be working great this morning!

[–] mrmanager 2 points 11 months ago

Yes it's really nice, and all my worries about disk space running out is gone. Now we enjoy Lemmy. :)

[–] Extrasvhx9he 1 points 11 months ago (1 children)

Wish you the best hope everything works out

[–] mrmanager 3 points 11 months ago (1 children)

Thank you. We are already up and migration is complete. It was faster than I expected!

Just looking around a bit now to verify that there is no weirdness... :)

[–] mrmanager 4 points 11 months ago