Hello, friends!

TL;DR: I am working on a new Lemmy frontend in nextJS. There is still much work to be done, but you can already have an early look at https://next.lemm.ee

First of all, quick note to lemm.ee users: I am making this announcement post in !meta@lemm.ee, as this is also a notice that I will be hosting an alternative frontend (lemmy-ui-next) for the first time on lemm.ee. Going forward, I will post updates about lemmy-ui-next in a separate dedicated community: !lemmy_ui_next@lemm.ee. If you're interested in future updates, please subscribe there!

What is lemmy-ui-next?

Lemmy is generally accessed through some kind of frontend UI. By default, Lemmy provides its own web interface (lemmy-ui), which you can find on the front page of most Lemmy instances (including lemm.ee). There are also several other independent frontends, for both the web and different mobile platforms, which I'm sure many of you are familiar with.

Lemmy-ui-next is a brand new alternative frontend, built from the ground up with modern and popular tooling - a framework known as NextJS. Lemmy-ui-next has (or aims to have) the following high-level features:

• Open source (AGPL)
• Drop-in replacement for lemmy-ui - same exact URL structure, so all existing links will continue working
• Very plain & minimalistic UI, strongly inspired by other link aggregator sites (of course including the original lemmy-ui!)
• Very basic and "typical" NextJS architecture, to encourage open source contributions
• Fully functional even when JavaScript is disabled (but works better with JS enabled!)
• Optimized data transfer between your browser and the server (filtering out only relevant data from the Lemmy API, caching, memoization)
• Strong focus on privacy and security (all authentication with the Lemmy API is done through secure httpOnly cookies, user IP addresses are not leaked to external image hosts, etc)

What is the current status of lemmy-ui-next?

I have mentally split the initial work I want to complete into 3 milestones:

1. Lurk - All read-only features of Lemmy
2. Participate - Voting/posting/commenting/DMs/reports, etc
3. Moderate - Handling reports, creating & managing communities, etc

I am now nearing completion of the first milestone. It's not 100% there yet, but you can already log in, browse, subscribe to communities and even vote. Some things may still look a bit wonky, and some features are still missing, but the core experience is getting there.

In terms of code contributions, I would ask anybody who is interested in getting involved to contact me first before working on anything. I am not looking for PRs just yet - the code structure is still a bit loose, and I am redefining it as I add more stuff. I would ideally really like to complete the first 3 milestones before opening things up for external contributors.

Who can use lemmy-ui-next?

At the moment, it is only hosted on this instance, at https://next.lemm.ee. I do not yet have any formal instructions for running it on other instances, but generally speaking, it is a simple NextJS app - to deploy it, you just need to do: npm install, npm run build and LEMMY_BACKEND=https://<your lemmy api here> npm run start.

Why not just improve lemmy-ui instead?

Lemmy-ui is an extremely important and valuable project. There has been a significant amount of hard effort put into it so far, and nobody can refute that it is the frontend which has really carried Lemmy to this point.

Unfortunately, there are some architectural problems with lemmy-ui (mostly related to how data is fetched and how sessions are stored in memory), all of which would require quite a significant rewrite to fix. Additionally, I think that the core technical solution used for lemmy-ui is just a bit too obscure, which has been an obstacle to my own contributions, as well as to contributions by others. If a rewrite is on the table anyway, then I believe a different technology is the best way forward.

Why not work on lemmy-ui-leptos instead?

Lemmy-ui-leptos is another rewrite of lemmy-ui, which is being lead by Lemmy maintainers. It is based around a Rust web framework called Leptos. I think this is really cool tech, and will be happy to host lemmy-ui-leptos on lemm.ee in the future as well.

There are a two key reasons why I personally decided to start working on another alternative, though:

• I have heard from several people on Lemmy that they feel like Leptos is a big barrier to entry in terms of them contributing
• Even for myself personally, I am very comfortable (and think I can move very fast) when working on something like NextJS, but with Leptos, I think the learning curve would be quite big and I would get much less done with any time I invest into it

My hope is that by providing a very vanilla alternative, I can provide an outlet for potential open source contributors who would like to work on Lemmy, but aren't prepared to do it with Leptos.

Does this mean that lemm.ee will change in ways I don't like?

First, let me be clear: lemm.ee will always host the default Lemmy frontend. This means lemmy-ui for now, and most likely lemmy-ui-leptos in the future.

I am however considering the possibility of switching things around at some point in the future, so that lemmy-ui-next will be hosted directly on lemm.ee, and lemmy-ui will be accessible on a different subdomain (like ui.lemm.ee). This would only happen once I have completed all 3 milestones for lemmy-ui-next. The main reason I am considering this is that I feel like I will always be in the best position to offer technical support to users on the frontend which I am myself maintaining. If you have any thoughts about this potential change, please let me know in the comments below!

That's about it for now!

This is something I've been thinking of doing for a while now, and I'm very excited to finally get the ball rolling! If you have a chance, please feel free to check out what https://next.lemm.ee looks like so far, and please let me know if you have any thoughts or feedback!

Hey

This is just a quick heads up that our host, Hetzner, has been experiencing networking issues today, which has caused some downtime for lemm.ee.

I have a workaround in place for now, so we should (fingers crossed) be recovering at the moment, but I am still waiting on the proper solution from Hetzner. You can track their issue here: https://status.hetzner.com/incident/9406c500-9c8b-48be-9591-a73691134096

Also, this is a good opportunity to remind everybody about https://status.lemm.ee - you can be sure that I will provide updates on that page as soon as I am aware of & dealing with any issues. I have been posting status updates for the current issue there as well.

Sorry for the inconvenience and I hope you have an otherwise great day!

UPDATE: Hetzner claims they have fixed the issue, but the problems have not been resolved for lemm.ee servers yet, so I am keeping my temporary workaround active for now. Will continue troubleshooting this tomorrow.

UPDATE 2: Hetzner has now fixed their issue, and our network has been restored to its original optimized state.

Hello!

I am sunaurus, the head admin of lemm.ee. Ever since I created my instance, I have been following a lot of public and private discussion channels between different parties involved with Lemmy. As I’m sure many others have also noticed, the discussions in such channels sometimes get heated, and in fact recently, I feel like there has been a constant trend in these discussions towards a lot of demands, hostility, negativity, and a general lack of empathy between different participants in the Lemmy network.

I am writing this post for a few reasons:

1. I would like add a bit of positivity by expressing my gratitude towards every single person who has helped improve Lemmy.
2. I want to speak up in defense of different people who have been receiving negativity lately.
3. There are a few false rumors spreading on Lemmy, which I would like to try and counteract with very simple evidence.
4. I want to remind everybody that at the end of the day, all of us care about building and improving Lemmy. We all have the same goal, and it’s too easy to lose sight of that.

I will split up what I want to say in this post by different user groups - users, mods, admins and developers. I understand that many people belong to several (or even all) of these groups, but I just want to highlight the value of, and express my gratitude to each group separately.

Users

At the end of the day, Lemmy would not be worth anything without the users. Users bring Lemmy to life by posting great content, getting involved in discussions in comments, helping surface interesting content for others through voting and even keeping the platform clean through reports. I am extremely thankful for all the users who have given me so much enjoyment on this platform.

I believe that users often get treated unfairly on Lemmy based on what instance they are participating from. I’m sure so many of you have noticed comments around Lemmy along the lines of “Oh, another user from , I’m going to completely ignore your stupid takes”. I’ve also many cases of people treating users as second-class citizen if they are not on the same instance - for example, I’ve seen users who are active and valuable participants in communities on another instance receive comments like “why are you participating in our discussions, go back to your own instance”. In my opinion this is completely counterproductive to the whole idea of federation. On a human level, I can understand it - you’re far more likely to notice or remember what instance somebody is posting from if you have a negative experience. As a result, as time goes by, people tend to develop negative views of each instance, despite potentially having had many positive interactions with other users of those same instances. The message I want to put out here is that instances, especially bigger ones, are not monoliths - do not judge users based on what instance they are browsing Lemmy from, judge them by their actual words and actions.

Mods

There are some excellent communities already on Lemmy, and these communities are all continuously being built up and maintained by mods. Mods put in huge amounts of their free time and energy in order to provide spaces for all Lemmy users. They form the first line of defense against bad actors, they keep communities alive and often receive no praise, only criticism. I am very grateful to everybody who has dedicated time to building communities on Lemmy.

Users rarely notice the lengths mods go to in order to keep communities running smoothly - mods more often than not only get noticed when users disagree with some mod actions. I believe mods deserve a lot better than this. Constructive criticism can of course be useful to improve communities, but it must be balanced with empathy and kindness towards people who have been putting in effort to provide something for users. Remember that there is another human being reading your words when you start writing about the mods of any particular community. Users who are not happy with mods of a certain community always have the opportunity to start their own community and run it as they like.

Admins

Admins provide two main key functions for the network:

1. Taking care of the actual infrastructure of Lemmy
2. Working as a higher level defense against bad actors, in cases where mods are not enough

I can tell from my own experience that being an admin of a bigger instance requires constant energy and attention. I don’t believe that there is a single medium-to-big instance where the admins have not put in hundreds (if not thousands) of hours of their free time, as well as in many cases, probably their own money. This is a service which admins provide for free, and it is necessary in order to keep the Lemmy network healthy. I have endless respect for anybody who is willing to put themselves in the position of a Lemmy admin.

I have seen awful messages towards admins from all the other groups listed here, including other admins. These messages range from condescending and rude, to downright hateful. I have seen admins treated as useless and their work taken for granted. I have seen people getting frustrated with admins for not spending every waking minute on Lemmy. I have seen some users consistently spreading provably false rumors about particular admins in an effort to tarnish their reputation on Lemmy.

Before you take out frustration on admins, please remember that they are also humans who have been working tirelessly to improve Lemmy in their own way.

Also, a reminder: the absolute best feature of Lemmy is that users are free to pick their instance - and as a result, users are also free to pick their admins. Even more than that, users can always become their own admins by spinning up their own instance. Yes, this requires dedication, effort, and research, but that’s exactly my point. It’s not easy running an instance, and mistreating people who do this as a free service is completely unacceptable.

Developers

Lemmy development has been lead by a few key maintainers, with a massive amount of smaller contributors. The software is constantly being improved at a very good pace, and everybody is able to benefit from this effort at no cost whatsoever. I am extremely grateful to everybody who has participated in the development of the Lemmy software, and other related software, as without you folks, none of us would even be here now.

There seems to be a huge amount of people with very little appreciation of the work that has gone into the software. I’m sure many of you have seen countless messages where people express that the devs should be doing more in one way or another. “They should work faster”, “they should prioritize this obviously most important feature”, “they should be available 24/7 to offer support”, etc. I just want to take a moment here and acknowledge what core maintainers have already done for Lemmy:

• Years worth of work on the code itself
• Offering support to the community and other admins
• Reviewing literally thousands of pull requests on GitHub
• Acting fast in stressful situations where the Lemmy network has been overloaded
• Not abandoning the project in the face of constant hateful users
• Sacrificing literally hundreds of thousands of euros in missed salaries which they could have been getting if they were working for a tech company instead of working on Lemmy

I also want to take this moment to discredit some rumors which I have seen repeated too many times:

1. Rumor: Lemmy devs do not accept outside code contributions

This is completely false - the maintainers are completely open to (and even constantly asking for) contributions. When somebody starts contributing, they will receive support and code reviews very quickly. I can tell you that I have experienced this myself several times, but that’s anecdotal, so let me also provide evidence:

a. Contributors list for the Lemmy backend: https://github.com/LemmyNet/lemmy/graphs/contributors

b. Contributors list for Lemmy UI: https://github.com/LemmyNet/lemmy-ui/graphs/contributors

Both of these lists include 100 different names, and that’s only because GitHub literally caps these pages to 100 users. Actually, the amount of different contributors is even bigger. If Lemmy devs did not accept and encourage outside contributions, then there would be no way for these lists to be so big.

2. Rumor: Lemmy devs work too slowly

This is an extremely entitled and frankly stupid claim. I try to keep on top of the changes made in the Lemmy repo, and let me tell you, the pace of improvement is very good.

I very firmly believe that if the network started downgrading to Lemmy versions from ~8 months ago, the whole network would just collapse, as none of the instances could keep up with the current volume. That is to say, we have come an extremely long way since last summer alone.

Let me provide some more evidence. Take a look at the Pulse page for the Lemmy backend on GitHub: https://github.com/LemmyNet/lemmy/pulse. As of writing this, Lemmy devs have merged 18 pull requests in the week leading up to this post - that’s an average of 2.5 merged PRs per day. This is extremely good for a project with a small underfunded team.

3. Rumor: Lemmy devs do not prioritize the important issues

There are two sides to this. First of all, there are endless users who turn to the Lemmy devs with what they believe is the most important issue and should immediately be prioritized - the problem is that almost none of these endless users have the same view of what the most important issue actually is! In that sense, it’s literally impossible to please everybody, because everybody wants different things.

On the other hand, even when Lemmy devs do prioritize things which some users have been desperately asking for, I have on several occasions seen a dismissive response along the lines of “too little too late”. Basically, the demands made are often unrealistic and impossible to meet.

If you are somebody who feels like Lemmy devs are not doing enough, I would ask you to please take a step back, look at the actual contributions which they have made, and consider how you yourself would feel if after making such a massive contribution, you would still need to listen to countless strangers on the internet tell you how you’re not good enough in their opinion.

Conclusion

Lastly, I am very thankful to anybody who took the time to read to the end of this post. Again, my goal is to try and defuse some of the hostility, as well as to put out a message of gratitude and positivity. I am very interested in the success of Lemmy as a whole, and that is much easier to achieve and maintain if we all work together. Thank you, I hope you're doing well, and have a nice weekend!

The RFC PR is here: https://github.com/LemmyNet/rfcs/pull/6

Reposting RFC contents below:

---

• Feature Name: report-inboxes
• Start Date: 2024-02-20
• RFC PR: LemmyNet/rfcs#0000
• Lemmy Issue: LemmyNet/lemmy#0000

Summary

Rather than combining all reports into a single report inbox, we should allow users to select whether they are reporting to mods or admins, and we should split reports into different inboxes based on that selection.

Motivation

The current approach has some shortcomings:

• Users are not currently able to bypass mods and report directly to admins - this may allow mods to conceal instance rule breaking in specific communities
• Admins are not aware of community rules, so they may wish to take no action for most community rule breaking reports. However, if an admin resolves such a report, the relevant community mods most likely never see it.
• Different instances may have different rules, but somebody resolving a report on one instance will resolve it for other instances as well, thus potentially resulting in missed reports.
• Mods might take local action on a report and mark it as resolved even in cases where a user should be banned from the entire instance. In this case, admins are very unlikely to see the report.

Guide-level explanation

When creating reports, users will be able to select if it's a mod report, or an admin report (or both)

Note: labels on the sreenshot are illustrative, actual labels can be more user-friendy. Maybe something like:

• Breaks community rules (report sent to moderators)
• Breaks instance rules (report sent to admins)

Instead of the current single report inbox, there will be three different kinds of inboxes

• Admin reports - show all reports sent to admins (only visible to admins)
• Mod reports - show all reports sent to mods for any communities the user moderates (visible to admins in case they are explicit mods in any communities)
  • This is equivalent to the report view that mods currently have in Lemmy already
• All reports - Shows a view of all (admin and mod) reports, only visible to admins
  • This is akin to the current 0.19.3 admin report view, and would allow admins to still keep an eye on mod actions on their instance if they wish

The UI wouldn't need to change for mods, but for admins, there would be a new selection at the top of the reports page (the "mod reports" tab would only be visible if the admin is also a mod in any community):

Resolving reports should be more granular

• Reports in the "admin reports" tab can only be manually resolved for admins of the local instance
  • To reduce overhead, banning the reported user on the user's home instance + removing reported content should automatically resolve reports for remote admins as well.
• Reports in the "mod reports" tab should be manually resolved by relevant mods (including admins, if they are explicit mods in the relevant community).
  • To reduce overhead, admins banning the reported user on the community instance OR the user's home instance + removing reported content should automatically resolve reports for mods as well
• Admins could still resolve reports in the "all reports" tab
  • If it's not an admin report, and not a mod report from a community the admin explicitly moderates, then there should be an additional warning/confirmation when resolving a report here. This is to prevent cases of admins accidentally preventing mods from moderating according to their own community rules.

To further clarify automatic resolution of reports: in any case where there is no further action possible, the report should be automatically resolved.

Mods should be able to escalate reports to admins

This would generate a corresponding report in the admin inbox.

Reference-level explanation

• In the UI, changes are needed for both reporting as well as the reports inbox views
• In the database and API, we should split reports by intended audience
• Federation needs to be changed as well in order to allow distinguishing the report target audience

Drawbacks

It might make reporting slightly more confusing for end users - the mod/admin distinction might not be fully clear to all.

Rationale and alternatives

Alternatively, we could make reporting even more granular. It would be possible to allow users to select only a specific instances admins as the intended report audience, for example. However, I think this has several downsides:

• Makes the report UI even more confusing
• Potentially takes away valuable information from other admins (imagine a user only reports CSAM to their own instances admins, while leaving the offending post authors home admins in the dark)

Prior art

Most other social networks allow users to select whether they are reporting a violation of community rules, or site rules as whole.

Unresolved questions

Does ActivityPub properly support splitting up reports like this?

Future possibilities

In the future, it might be a nice addition to have some automation to always escalate to admins, even if they're submitted as mod reports, based on report keywords. For example, "CSAM", "Spam", etc.

Hey folks

Some of you may have noticed comments complaining about spam and lack of moderation within the past day or so. Maybe you've even noticed a few spam posts yourself (hopefully not too much, as we have automations in place on lemm.ee to remove the spam as soon as it is posted).

I just wanted to write a quick post with some context about the attack, what we are doing about it, and how you can help.

Context

Allegedly, a group of kids in Japan have created a bot, which signs up on different Fediverse instances and posts spam into different communities. The spam generally consists of Japanese text and/or an image and/or a bunch of random @mentions into different communities. You can check a post on Mastodon with more information here: https://mastodon.de/@ErikUden/111940301222380638

What we are doing about it

Many instances are actively working to limit this spam-wave, and lemm.ee is no different. Thankfully, we have not had to deal with any bot sign-ups on our instance (potentially as a result of different protections we have implemented for sign-ups), but we still suffer the effects of the spam, even if it's posted from other instances. To help us quickly eliminate most of the spam for lemm.ee users, I am continually tuning our @adminbot to automatically detect and remove content posted in this current spam-wave.

We cannot remove content from the wider Fediverse if it's not posted there by a lemm.ee user, so our automated removals won't help users on other instances, but we are at least improving the experience for our own users. For an example, you can compare how /c/opensource@lemmy.ml currently looks like on lemm.ee, to how it looks like on this screenshot I took from another smaller instance:

How you can help

First and foremost, please continue reporting any spam you find, so that relevant mods and admins can deal with it. I am very grateful to users who help us identify spam through reports, and your reports are precisely what allow me to implement automated content removal for more extreme spam-waves such as this current one.

Secondly, I am seeking for a few volunteers to grow the lemm.ee admin team. I am purposely burying this at the bottom of the post, to hopefully pre-filter out some candidates who would want to join for the wrong reasons. If you have read until this point in the post, then I assume you are already quite interested in improving the experience on lemm.ee, so if you feel like you could contribute to the admin team, please read on.

First, I will say a few words about who we are looking for, then I will describe what kind of tasks you would have as an admin, and finally, I will cover some significant downsides of joining the admin team.

We are looking for folks who more or less match the following profile:

• You have already been active on the Fediverse for several months (not necessarily on lemm.ee)
• Previous mod experience would be a huge plus
• You should feel a strong agreement with our basic instance rules and our administration & federation policy
• You should be prepared to be exposed to some vile content through reports
• You are OK with using Discord as the main method of admin communication (that is what we have settled on and will continue using for the foreseeable future)

As volunteers, we don't expect admins to be available 24/7, but as our instance grows, I do think it would be quite important to achieve a state of pretty good timezone coverage with our admin team, so please only consider applying if you are already regularly active on Lemmy.

As for what tasks admins are responsible are for: it's mostly covered in the administration policy post linked above. But in short, you should be prepared to regularly check the report queue, contact users with friendly messages to de-escalate conflicts, issue bans, remove content, and monitor the activity of @adminbot. Additionally, if you're interested in taking a more hands-on approach to any kind of community-building on lemm.ee, then this would be totally welcome as well, but not strictly considered a core responsibility for admins.

Please note that the lemm.ee admin team has an absolute zero tolerance policy against any kind of abuse towards minority communities. If you do not share this mindset, then please do not consider applying.

Finally, let me share some negative aspects about joining the admin team. I think this will probably reduce the amount of any potential candidates, but I still feel it's important to be honest and upfront about this:

Through the report queue, you will regularly see absolutely vile content which you might otherwise never even notice on Lemmy. Many users come to Lemmy to spread hate, post disturbing images, etc, and in order to clean such content up for other users, mods and admins need to actually be exposed to this content in much larger amounts than regular users.

Additionally, while Lemmy is constantly being improved by the developers, the moderation tools are still quite rough around the edges. Lemmy is not at 1.0 yet, and that will most likely become even more obvious to you as you work on admin tasks.

Maybe this is the most important one: no matter what you do, there will always be people unhappy with how you apply our rules. I have seen countless comments complaining about lemm.ee admins specifically. I have been told by complete strangers that they hate me. I have seen many complaints about us moderating too harshly. I have seen complaints about us not moderating enough. I have seen users on Lemmy make up wild stories about our admin team, and share them as facts. There are of course plenty of supportive users, but the negative experiences tend to leave a much more lasting impression.

If after reading all of the above, you are still motivated to help make lemm.ee a better place through offering your help in the admin team, please contact me on Discord (@sunaurus)!

That's all from me for now. Thank you very much to anybody who went through this whole wall of text, and I hope you are all having a good weekend!

Hey folks!

Just a quick update: we now have a dedicated status page for lemm.ee.

You can find it at status.lemm.ee. It currently contains three sections:

1. A web status section, which I will update manually to communicate issues about lemm.ee
2. A financial status section, which I will update monthly to give an overview of how we're doing financially
3. A federation section, which automatically checks the current federation status, both incoming and outgoing, between lemm.ee and other instances. By default it shows 3 large instances, but you can also search for any specific instance you are interested in.

This status page is hosted completely separately from our main servers, so if there is any trouble with our servers, you can expect the status page to still be available!

If you have any issues with this page, or any other thoughts, feel free to comment.

Hey folks

This is a heads up that I will be performing some maintenance and hardware upgrades on our database this Saturday.

We are currently experiencing several spikes throughout the day which cause our database to become overloaded - this results in degraded performance for many users. The spikes are happening due to a combination of continued growth of the database, some expensive periodic scheduled tasks which Lemmy runs, and fluctuating traffic patterns. Some of this can be optimized on the code level in the future, but it seems that the best way to deal with it right now is to add some additional resources to our database server.

I am intending to switch to slightly different hardware in this upgrade, and will be unable to make this switch without downtime, so unfortunately lemm.ee will be unavailable for the duration.

As our database has grown quite a bit, cloning it will most likely take a few hours, so I expect the downtime to last 2-3 hours. Sorry for the inconvenience, I am hopeful that it will be worth it and that this upgrade will significantly reduce some of our recent long page load times!

---

Edit: upgrade complete!

I have now migrated the lemm.ee database from the original DigitalOcean managed database service to a dedicated server on Hetzner.

As part of this migration, I have also moved all of our Lemmy servers from the DigitalOcean cloud to Hetzner's Cloud. I always want the servers to be as close as possible to the database, in order to keep latencies low. At the same time, I am very interested in having the ability to dynamically spin up and down servers as needed, so a cloud-type solution is really ideal for that. Fortunately, Hetzner allows connecting cloud servers to their dedicated servers through a private network, so we are able to take advantage of a powerful dedicated server for the database, while retaining the flexibility of the cloud approach for the rest of our servers. I'm really happy with the solution now.

In terms of results, I am already seeing far better page load times and far less resource use on the new hardware, so I think the migration has been a success. I will keep monitoring things and tuning as necessary.

Happy new year!

Hi folks! I hope everybody had a good holiday period and I wish you all the best for 2024. I have some quick updates to share about lemm.ee:

Image uploads

Image uploads are now enabled for all lemm.ee users 4 weeks after account creation. The upload size limit is currently set to 500kb.

The 4 week account age requirement is in place to discourage spam and abuse. It is of course not a fool-proof solution, but let's give it a go and see what the results are.

Please note that lemm.ee is not intended to be a image hosting service! Feel free to upload avatars and banners for your profile and communities, but please be aware that we reserve the right to modify the upload limits going forward, as well as delete old images if storage costs become too high.

For image posts and comments, it would still be preferable for you to use an external image hosting service.

Federation delays

Over the holidays, our outgoing federation workers began experiencing some significant delays. I have been working on this problem for the past few days, and after updating to 0.19.1, applying some additional patches to the code, and changing our infrastructure a bit, I believe the issue has been resolved.

The good news is that now that we are on 0.19, problems such as this do not cause Lemmy to completely drop federated activities, as we now retain a persistent queue of federation activities for all linked instances. This means that after the issue was resolved, our federation workers started going through the backlog of likes, comments, and posts which you had made over the past several days, and sending these out to other instances. Essentially, all of your activities did end up reaching their target servers, just with some additional delay.

One quick side-note here, while we are now federating your activities in real-time again to most big instances, there is still a bit of a backlog left on the lemm.ee -> lemmy.world federation (it is a few days behind). I expect this to also catch up by tomorrow.

Performance

The new persistent federation queue is still quite a new feature in Lemmy, so it's a bit rough around the edges - after resolving the federation issues, our federation workers started going through the queue at extreme speed, which caused intense additional load on our database. This was one of the reasons for some performance degradation many of you noticed over the past few days.

Additionally, since updating to 0.19, there have been regular performance issues for many users. I have managed to solve a few of these by making some changes in our infrastructure, but I am also aware of a few more issues which I will continue to monitor and hopefully improve in the near future. Sorry for the inconvenience, I hope that the changes I have made so far will help make it a bit smoother already!

That's all from me for now, as always, feel free to comment if you have any thoughts, and have a nice day!

Hey folks!

Lemmy 0.19 was released this week! It brings a bunch of awesome new features, so I hope you are all willing to forgive some downtime in order to upgrade to this latest version.

Unfortunately some migration will be necessary as part of this upgrade, so it might take a while, but I will try to keep it as short as possible.

I hope you are all having a great holiday period, and I will see you soon in 0.19!

---

Edit: Update complete!

Welcome to 0.19! Unfortunately, the upgrade took somewhat longer than usual, but I believe everything is in order now. As always, please let me know if you notice anything strange, and have fun!

Hey folks!

I made a short post last night explaining why image uploads had been disabled. This was in the middle of the night for me, so I did not have time to go into a lot of detail, but I'm writing a more detailed post now to clear up where we are now and where we plan to go.

What's the problem?

As shared by the lemmy.world team, over the past few days, some people have been spamming one of their communities with CSAM images. Lemmy has been attacked in various ways before, but this is clearly on a whole new level of depravity, as it's first and foremost an attack on actual victims of child abuse, in addition to being an attack on the users and admins on Lemmy.

What's the solution?

I am putting together a plan, both for the short term and for the longer term, to combat and prevent such content from ever reaching lemm.ee servers.

For the immediate future, I am taking the following steps:

1) Image uploads are completely disabled for all users

This is a drastic measure, and I am aware that it's the opposite of what many of our users have been hoping, but at the moment, we simply don't have the necessary tools to safely handle uploaded images.

2) All images which have federated in from other instances will be deleted from our servers, without any exception

At this point, we have millions of such images, and I am planning to just indiscriminately purge all of them. Posts from other instances will not be broken after the deletion, the deleted images will simply be loaded directly from other instances.

3) I will apply a small patch to the Lemmy backend running on lemm.ee to prevent images from other instances from being downloaded to our servers

Lemmy has always loaded some images directly from other servers, while saving other images locally to serve directly. I am eliminating the second option for the time being, forcing all images uploaded on external instances to always be loaded from those servers. This will somewhat increase the amount of servers which users will fetch images from when opening lemm.ee, which certainly has downsides, but I believe this is preferable to opening up our servers to potentially illegal content.

For the longer term, I have some further ideas:

4) Invite-based registrations

I believe that one of the best ways to effectively combat spam and malicious users is to implement an invite system on Lemmy. I have wanted to work on such a system ever since I first set up this instance, but real life and other things have been getting in the way, so I haven't had a chance. However, with the current situation, I believe this feature is more important then ever, and I'm very hopeful I will be able to make time to work on it very soon.

My idea would be to grant our users a few invites, which would replenish every month if used. An invite will be required to sign up on lemm.ee after that point. The system will keep track of the invite hierarchy, and in extreme cases (such as spambot sign-ups), inviters may be held responsible for rule breaking users they have invited.

While this will certainly create a barrier of entry to signing up on lemm.ee, we are already one of the biggest instances, and I think at this point, such a barrier will do more good than harm.

5) Account requirements for specific activities

This is something that many admins and mods have been discussing for a while now, and I believe it would be an important feature for lemm.ee as well. Essentially, I would like to limit certain activities to users which meet specific requirements (maybe account age, amount of comments, etc). These activities might include things like image uploads, community creation, perhaps even private messages.

This could in theory limit creation of new accounts just to break rules (or laws).

6) Automated ML based NSFW scanning for all uploaded images

I think it makes sense to apply automatic scanning on all images before we save them on our servers, and if it's flagged as NSFW, then we don't accept the upload. While machine learning is not 100% accurate and will produce false positives, I believe this is a trade-off that we simply need to accept at this point. Not only will this help against any potential CSAM, it will also help us better enforce our "no pornography" rule.

This would potentially also allow us to resume caching images from other instances, which will improve both performance and privacy on lemm.ee.

---

With all of the above in place, I believe we will be able to re-enable image uploads with a much higher degree of safety. Of course, most of these ideas come with some significant downsides, but please keep in mind that users posting CSAM present an existential threat to Lemmy (in addition to just being absolutely morally disgusting and actively harmful to the victims of the abuse). If the choice is between having a Lemmy instance with some restrictions, or not having a Lemmy instance at all, then I think the restrictions are the better option.

I also would appreciate your patience in this matter, as all of the long term plans require additional development, and while this is currently a high priority issue for all Lemmy admins, we are all still volunteers and do not have the freedom to dedicate huge amounts of hours to working on new features.

---

As always, your feedback and thoughts are appreciated, so please feel free to leave a comment if you disagree with any of the plans or if you have any suggestions on how to improve them.

Sorry for the short post, I'm not able to make it nice with full context at the moment, but I want to quickly get this announcement out to prevent confusion:

Unfortunately, people are uploading child sexual abuse images on some instances (apparently as a form of attack against Lemmy). I am taking some steps to prevent such content from making it onto lemm.ee servers. As one preventative measure, I am disabling all image uploads on lemm.ee until further notice - this is to ensure that lemm.ee can not be used as gateway to spread CSAM into the network.

It will not possible to upload any new avatars or banners while this limit is in effect.

I'm really sorry for the disruption, it's a necessary trade-off for now until we figure out the way forward.