this post was submitted on 04 Jul 2023
552 points (99.5% liked)

Lemmy.World Announcements

28968 readers
3 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages 🔥

https://status.lemmy.world

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations 💗

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 1 year ago
MODERATORS
ruud@lemmy.world
lwadmin@lemmy.world
lwCET@lemmy.world
jelloeater85@lemmy.world
Serinus@lemmy.world
552
Lemmy.world status update 2023-07-04 (lemmy.world)
submitted 1 year ago* (last edited 1 year ago) by ruud@lemmy.world to c/lemmyworld@lemmy.world
238 comments fedilink hide all child comments
 

Status update July 4th

Just wanted to let you know where we are with Lemmy.world.

Issues

As you might have noticed, things still won't work as desired.. we see several issues:

Performance

  • Loading is mostly OK, but sometimes things take forever
  • We (and you) see many 502 errors, resulting in empty pages etc.
  • System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)

Bugs

  • Replying to a DM doesn't seem to work. When hitting reply, you get a box with the original message which you can edit and save (which does nothing)
  • 2FA seems to be a problem for many people. It doesn't always work as expected.

Troubleshooting

We have many people helping us, with (site) moderation, sysadmin, troubleshooting, advise etc. There currently are 25 people in our Discord, including admins of other servers. In the Sysadmin channel we are with 8 people. We do troubleshooting sessions with these, and sometimes others. One of the Lemmy devs, @nutomic@lemmy.ml is also helping with current issues.

So, all is not yet running smoothly as we hoped, but with all this help we'll surely get there! Also thank you all for the donations, this helps giving the possibility to use the hardware and tools needed to keep Lemmy.world running!

you are viewing a single comment's thread
view the rest of the comments
[–] Kalcifer@lemmy.world 17 points 1 year ago* (last edited 1 year ago) (3 children)

That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%

Lemmy has a memory leak? Or, should I say, a "lemmory leak"?

[–] Sigmatics@lemmy.ca 6 points 1 year ago (1 children)

A pretty bad one at that...

[–] SomeOtherUsername@lemmynsfw.com 5 points 1 year ago (4 children)

But... but... Rust...

[–] ObviouslyNotBanana@lemmy.world 10 points 1 year ago

Rust makes holes and that's how leaks happen

[–] donalonzo@lemmy.world 2 points 1 year ago

Rust protects you from segfaulting and trying to access deallocated memory, but doesn't protect you from just deciding to keep everything in memory. That's a design choice. The original developers probably didn't expect such a deluge of users.

[–] bad3r@lemmy.one 1 points 1 year ago* (last edited 1 year ago)

Leaking memory is safe

Rust’s memory safety guarantees make it difficult, but not impossible, to accidentally create memory that is never cleaned up (known as a memory leak). Preventing memory leaks entirely is not one of Rust’s guarantees in the same way that disallowing data races at compile time is, meaning memory leaks are memory safe in Rust. We can see that Rust allows memory leaks by using Rc and RefCell: it’s possible to create references where items refer to each other in a cycle. This creates memory leaks because the reference count of each item in the cycle will never reach 0, and the values will never be dropped.

[–] SomeOtherUsername@lemmynsfw.com 0 points 1 year ago (1 children)

I’m calling it - if there’s actually a memory leak in the Rust code, it’s gonna be the in memory queues because the DB’s iops can’t cope with the number of users.

[–] SomeOtherUsername@lemmynsfw.com 7 points 1 year ago* (last edited 1 year ago) (2 children)

I think I found what eats the memory. DB iops isn't the cause - looks like the server doesn't reply before all the database operations are done. The problem is the unbounded queue in the activitypub_federation crate, spawned when creating the ActivityQueue struct. The point is, this queue holds all the "activities" - events to be sent to federated instances. If, for whatever reason, the events aren't delivered to all the federated servers, they are retried with an exponential backoff for up to 2.5 days. If even a single federated instance is unreachable, all events remain in memory. For a large instance, this will eat up the memory for every upvote/downvote, post or comment.

Lemmy needs to figure out a scalable eventual consistency algorithm. Most importantly, to store the messages in the DB, not in memory.

[–] tool@r.rosettast0ned.com 3 points 1 year ago (1 children)

You should take this entire comment and paste it in as a issue on the Lemmy Github Issues page.

[–] SomeOtherUsername@lemmynsfw.com 2 points 1 year ago

I think the devs have been aware of the issue, theoretically, for a while. A proper solution requires some significant changes, so it was being postponed because this wasn't considered urgent.

[–] Coelacanth@lemmy.world 3 points 1 year ago

You should consider bringing this up at !lemmyperformance@lemmy.ml

[–] grue@lemmy.world 3 points 1 year ago

Lemmy has a memory leak? Or, should I say, a “lemmory leak”?

A lemmory meek, obviously!

[–] 257m@lemmy.ml 1 points 1 year ago (1 children)

Wait isn't lemmy written in rust how do you create a memory leak in rust? Unsafe mode?

[–] donalonzo@lemmy.world 2 points 1 year ago* (last edited 1 year ago) (1 children) 
let mut memory_leak = Vec::<u8>::new();
loop {
  memory_leak.push(0);
}
[–] meisme@lemmy.world 2 points 1 year ago (1 children)

That's not a memory leak though. That's just hording memory. Leaked memory is inaccessible.

[–] preciouspupp@sopuli.xyz 1 points 1 year ago (1 children)

In the example it’s inaccessible.

[–] meisme@lemmy.world 3 points 1 year ago (1 children)

It's not. The vec is still accessible and if it goes out of scope rust automatically clears the memory.

[–] crate_of_mice@lemmy.world 2 points 1 year ago (2 children)

But it will never fall out of scope because of the loop

[–] meisme@lemmy.world 4 points 1 year ago (1 children)

Exactly but it's still accessible since it's in scope.

[–] crate_of_mice@lemmy.world 1 points 1 year ago (1 children)

Right but where does it say that something has to be inaccessible to qualify as a memory leak?

[–] meisme@lemmy.world 1 points 1 year ago (1 children)

"A memory leak is a process in which a program or application persistently retains a computer’s primary memory. It occurs when the resident memory program does not return or release allocated memory space, even after execution, resulting in slower or unresponsive system behavior." Source

[–] preciouspupp@sopuli.xyz 1 points 1 year ago* (last edited 1 year ago)

That’s not a serious source. Any unbound allocation is a memory leak if it serves no useful purpose.