this post was submitted on 31 Jul 2023
282 points (99.6% liked)

Lemmy.World Announcements

30521 readers
1 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages πŸ”₯

https://status.lemmy.world/

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations πŸ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 2 years ago
MODERATORS
 

Lemmy.world has been down between 02:00 UTC and 05:45 UTC. This was caused by the database spiking to 100% cpu (all 32 cores/64 threads!) due to inefficient queries been fired to the db very often.

I’ve collected the logs and we’ll be checking how to prevent this. (And what caused this)

top 38 comments
sorted by: hot top controversial new old
[–] zikk_transport2@lemmy.world 96 points 2 years ago (3 children)

lemmy.world is a test environment for Lemmy developers.. πŸ˜… Jokes aside, issue is an issue.

[–] slimerancher@lemmy.world 40 points 2 years ago

Well, as pretty much the biggest instance, it provides the best data for load-testing. 🫣

[–] ReadyUser31@lemmy.world 35 points 2 years ago

All Lemmy instances are test environments right now. It's just that lemmy.world is being tested the hardest.

[–] Blaze@sopuli.xyz 22 points 2 years ago (1 children)

It's a good reminder that other large instances (lemm.ee, sh.itjust.works, reddthat.com, sopuli.xyz) usually offer higher uptime while showing the same content as LW

[–] ReadyUser31@lemmy.world 12 points 2 years ago (1 children)

My issue with the smaller instances is that I enjoy browsing Local, and this is typically not as active or interesting on there. Is there a way to browse lemmy.world's Local from another instance?

[–] Blaze@sopuli.xyz 1 points 2 years ago (2 children)

Local makes sense for a country or field focused instance (queer rights, programming, even porn or whatever), not so much for generalist instances, unfortunately.

The way you use local only makes sense now as most of the communities are on LW. Once a few of them start to branch out (such as lemdro.id or startrek.website), if you want to see that content, you will either subscribe or use all

[–] BURN@lemmy.world 4 points 2 years ago

Honestly I really don’t like that model. I only subscribe to communities I am active in and all is just porn spam. Local is 100% the best way to browse right now.

[–] luis123456@mastodon.social 2 points 2 years ago

@Blaze @ReadyUser31 I love using all to find new stuff

[–] flubba86@lemmy.world 60 points 2 years ago (2 children)

Every Lemmy update:

"We fixed some performance issues by optimising some queries."

Also: "To balance it out, we added some new even more inefficient queries."

[–] kratoz29@lemmy.world 39 points 2 years ago (1 children)
[–] Rooki@lemmy.world 5 points 2 years ago (1 children)

Next time just god damn upload it directly. Thx

[–] kratoz29@lemmy.world 5 points 2 years ago (1 children)
[–] Rooki@lemmy.world 1 points 2 years ago (1 children)

what You can just simply copy the video directly and paste it directly into the comments

[–] kratoz29@lemmy.world 12 points 2 years ago (1 children)

I don't know how to do that from Android and into the Summit app.

[–] Rooki@lemmy.world 2 points 2 years ago (1 children)

There should be an icon like this:

And you can probably upload a picture there

[–] kratoz29@lemmy.world 3 points 2 years ago (1 children)

There is, but my file was a video, maybe that's why I could not.

[–] Rooki@lemmy.world 1 points 2 years ago (1 children)

Videos can be uploaded. Someone uploaded a full video of the simpsons.

[–] kratoz29@lemmy.world 2 points 2 years ago

Well, I just tried again, now from Voyager and Connect for Lemmy and no dice.

May I see that full video post?

Maybe this is allowed in the desktop website or instance.

[–] ElBarto@lzrprt.sbs 8 points 2 years ago (1 children)

How else are they supposed to patch things out? can't leave it up to fate, gotta make your own destiny!

[–] kratoz29@lemmy.world 2 points 2 years ago

Huh, isn't this instance only?

I just think tried from Connect for Lemmy in this comment and I couldn't.

[–] ttmrichter@lemmy.world 31 points 2 years ago (2 children)

(And what caused this)

Prediction: bad database programming. ;)

[–] NewNewAccount@lemmy.world 14 points 2 years ago (1 children)

Are we extracting enough value out of our volunteer developers and DBAs?!

[–] ewe@lemmy.world 7 points 2 years ago

The beatings will continue until morale improves!

[–] MrPoopyButthole@lemmy.world 7 points 2 years ago

Or clever denial of service attacks

[–] Blaze@sopuli.xyz 21 points 2 years ago

Thank you for the update!

[–] Possible_EmuWrangler@lemmy.world 17 points 2 years ago* (last edited 2 years ago)

Thanks for keeping us updated. FYI i noticed an issue there was an error message saying to check (the matix) and (somewhere else, Lemmy community support?). Both of them pointed to the same URL, but im sure they were meant to point to different places.

Edit. Happed again and I took notes. Both point to lemmy.ml community support

[–] AlmightySnoo@lemmy.world 8 points 2 years ago (1 children)

It seems to start at roughly the same hour every day, at around 01:20 UTC

[–] ruud@lemmy.world 10 points 2 years ago

Yup. We might be on to something now

[–] Yoz@lemmy.world 7 points 2 years ago (2 children)

People move to smaller instances so that with such outage not everyone is affected. Use fediverse as its supposed to be used.

[–] Spacebar@lemmy.world 17 points 2 years ago (2 children)

What's the name of the server you are running?

A large instance today will be a small instance in the future. There are hardly any users on lemmy compared to other more established platforms. So if lemmy is to ever handle a lot more users, stress testing the code makes a lot of sense.

What's going to happen in the future, do you expect there to be 50,000 servers? That's unrealistic.

[–] Sl00k@programming.dev 3 points 2 years ago (1 children)

Instances should be divided more into groups of Communities. So they theoretically don't grow infinitely, only as high as the "group" if communities grows. Ex. An NBA or Sports instance containing /c/NBA /c/NFL /c/NHL and all the related teams. Or similar to the programming.dev instance all being programming and development. While these would grow it would grow at a much slower rate than everything in one instance and be much more maintainable.

Of course this is somewhat of a social construct so everyone has to be in agreement with how to handle this and move accordingly, which won't happen.

Long-term long-term for federation there has to be a distributed computing solution that allows the users to contribute to hosting.

[–] ruffsl@programming.dev 2 points 2 years ago

Ex. An NBA or Sports instance containing /c/NBA /c/NFL /c/NHL and all the related teams.

[–] Yoz@lemmy.world 1 points 2 years ago (1 children)

You're not taking into account that some people are dumb as fuck. They will sit on one instance and when the instance goes down , they'll start whining

[–] xtremeownage@lemmyonline.com 12 points 2 years ago

They will sit on one instance and when the instance goes down , they’ll start whining

Its true. Especially so, since its my instance, and it being broken means I need to fix it. :'(

[–] cerevant@lemmy.world 10 points 2 years ago (2 children)

I can’t claim to know what the designers intended, but having users spread across a large numbers of servers is terribly inefficient for how Lemmy works: each server maintains a copy of each community that it’s users are subscribed to, and changes to those communities need to be communicated across each of those instances.

Given this architecture, it is much more efficient and robust to have users concentrate on what are effectively high performance cacheing servers, and communities spread out on smaller, interest focused instances.

[–] ewe@lemmy.world 4 points 2 years ago

Yeah, I think this is the way things should move in the future. Have community vs user focuses on servers instead of having the same server get hit with both high community/comment usage and a server with lots of login/audit/user browsing requests. Servers with big communities could focus on stability and perfomance. Servers with users could focus on cool UIs and features for their users.

[–] Yoz@lemmy.world 1 points 2 years ago

Bro I was talking about you. read my previous comment

[–] md5crypto@lemmy.world 1 points 2 years ago

Another DDOS attack?