211
submitted 1 year ago* (last edited 1 year ago) by th3raid0r@tucson.social to c/technology@beehaw.org

Look, we can debate the proper and private way to do Captchas all day, but if we remove the existing implementation we will be plunged into a world of hurt.

I run tucson.social - a tiny instance with barely any users and I find myself really ticked off at other Admin's abdication of duty when it comes to engaging with the developers.

For all the Fediverse discussion on this, where are the github issue comments? Where is our attempt to convince the devs in this.

No, seriously WHERE ARE THEY?

Oh, you think that just because an "Issue" exists to bring back Captchas is the best you can do?

NO it is not the best we can do, we need to be applying some pressure to the developers here and that requires EVERYONE to do their part.

The Devs can't make Lemmy an awesome place for us if us admins refuse to meaningfully engage with the project and provide feedback on crucial things like this.

So are you an admin? If so, we need more comments here: https://github.com/LemmyNet/lemmy/issues/3200

We need to make it VERY clear that Captcha is required before v0.18's release. Not after when we'll all be scrambling...

EDIT: To be clear I'm talking to all instance admins, not just Beehaw's.

UPDATE: Our voices were heard! https://github.com/LemmyNet/lemmy/issues/3200#issuecomment-1600505757

The important part was that this was a decision to re-implement the old (if imperfect) solution in time for the upcoming release. mCaptcha and better techs are indeed the better solution, but at least we won't make ourselves more vulnerable at this critical juncture.

top 50 comments
sorted by: hot top controversial new old
[-] Gaywallet@beehaw.org 65 points 1 year ago* (last edited 1 year ago)

I find it reasonably amusing that many people's solutions seem to be "just defederate bro". As in if this conversation isn't happening on an instance which chose to defederate and received thousands of negative comments, from other instances, about this choice. We're still being harassed by users, all over our instance, who are unhappy with this.

I also find it amusing that many people say the solution is to build your own solution. Do you not want the fediverse to grow? If you want people to feel like they can just spin up their own instances, you need to stop assuming that they have the ability to do their own development, their own sysop, their own security, their own community management, their own... everything. People are not omniscient and the outright hostility towards someone asking for help, or surfacing their opinion on the matter isn't helping.

Without adequate tools, I don't see how most instances aren't driven towards simply existing on their own. Large instances need tools to deal with malicious actors, as they are the targets. The solution to defederate ignores the ability for people to just spin up new instances, to hijack existing small instances with less resources for security, sysops, to watch/manage their DB, to prevent malicious actors. I've already seen proposed solutions which involve scraping for all instances with less than a certain number of users to defederate on principle (inactive, too many users/post ratio). We're fighting spam bots right now, who are targeting instances which don't have captcha enabled.

Follow this thinking through to it's conclusion. If the solution is to defederate, and there are potentially unlimited attack vectors, what must a large instance do to not overburden its resources? Switch from blacklist to whitelist? Defederate from all small instances? How is this sustainable for the fediverse? If you want people to be interacting with each other, you need to provide the tools for this to happen in the presence of malicious actors. You can't just assume these malicious actors won't exist, or will just overcome any and all obstacles you throw in their way because you're smart enough to understand how to bypass captcha or other issues.

This isn't just an issue of whether captcha or some other anti-spam measure is used, it's an issue about the overall health of the fediverse. Please think wider about the impact before offering your 2c about how captchas are worthless or how you hate cloudflare. I don't think the user that posted this cares about the soapbox you want to preach from- they're looking for solutions.

[-] alyaza@beehaw.org 25 points 1 year ago* (last edited 1 year ago)

I also find it amusing that many people say the solution is to build your own solution. Do you not want the fediverse to grow? If you want people to feel like they can just spin up their own instances, you need to stop assuming that they have the ability to do their own development, their own sysop and sysad, their own security, their own community management, their own… everything. People are not omniscient and the outright hostility towards someone asking for help, or surfacing their opinion on the matter isn’t helping.

to underscore this: if we had to do all of this this instance would not exist and/or we would have shut off applications about 10,000 people ago. we do not have the capabilities to do all of this even now with like a dozen people volunteering to help us! we are one of the largest instances on Lemmy and one of the most active! please recognize how ridiculous and burdensome it is to just throw more non-inbuilt tech at problems like this, and how exclusionary that is going to be to anybody who is without free time and extremely tech-savviness. if you want this space to grow it needs to be at a point where people can just use it and not have to worry about this shit.

[-] th3raid0r@tucson.social 23 points 1 year ago* (last edited 1 year ago)

I'm a DevOps/SysOps/SecOps engineer - have been for over a decade now. Even if I CAN do all the things listed, it takes time to do it. It takes time to configure your networking layer, especially when documentation of the underlying app is in flux and never 100% correct. It takes time to secure your server, especially when the "prod" configuration in the repo isn't really that secure at all.

Folks saying to just "code it myself" - sure, let me stop doing my day job and start planning on this completely unpaid enhancement. Let me tell my wife - "Sorry babe, gotta prove this internet person wrong and it must be today - can't go to board game night with you". I mean, I'll actually likely end up coding it myself, but when I can. Not when the trolls who say "Oh, come on, it'll be EZ" - yeah, I know better than that.

Folks just say to "Use other solutions" - Great! I already budgeted 150/month of my own money. Oh wait, that doesn't matter much when I have to worry about instances that can't spend that type of scratch.

[-] towerful@beehaw.org 7 points 1 year ago

The 2 Lemmy devs have funding. About 1500 total from community support, with the rest coming from a sponsorship/incubator type deal. A deal which pays out when targets/goals are achieved.
Which made me laugh at this:

sure, let me stop doing my day job and start planning on this completely unpaid enhancement

Which is entirely what you are asking the Lemmy devs to do.

Thanks for raising awareness of the spam-bot-account issue.

load more comments (2 replies)
[-] th3raid0r@tucson.social 17 points 1 year ago

Personally, I find it reasonably amusing that defending an open source, arguably collectivist project requires appeals to individualism.

"You can build it" "Just defederate" "It's the instance owner's responsibility" "You can do X for your instance, its in your control"

Like, which is it? Is this a collective undertaking by a community of multiple stakeholders or is this the Dev's individual project and they don't have to listen to anyone?

[-] DrWeevilJammer@lm.rdbt.no 9 points 1 year ago

Is this a collective undertaking by a community of multiple stakeholders or is this the Dev's individual project and they don't have to listen to anyone?

Devs, especially extremely busy ones "listen" via pull requests. Instead of badgering the devs, put together some devs of your own, get some code working, and submit it as a PR.

If they don't accept it, you now have code that does what you want, and it would be easy to create your own fork.

[-] th3raid0r@tucson.social 13 points 1 year ago

Yeah, and this would work fine for new features. But for removing existing features that alter the entire ecosystem regardless if you upgrade or not? This isn't at all the same, and casting it as such isn't honest.

I feel like folks keep making this a technical merit discussion when that's not at all what it is. A better technical solution is required, I agree. I'm not even disagreeing that captcha can be bypassed - but so can a lock, or a door, or any security feature really given a sufficiently intelligent threat.

But so far the captcha has already made some difference in what instances have spam account problems and those that don't. To argue that it isn't perfect is a logical fallacy that's making my head hurt. Shall we get rid of door locks because they can be picked? Should we get rid of garage doors entirely with the new hacking devices available - obviously the security isn't perfect so why have it at all?

Since when did perfect become the enemy of good? We had a good solution... And now we're throwing it out of a better one, fine! But leave the good one in place until then.

load more comments (1 replies)
[-] freeman@lemmy.pub 9 points 1 year ago

I’ve already seen proposed solutions which involve scraping for all instances with less than a certain number of users to defederate on principle (inactive, too many users/post ratio). We’re fighting spam bots right now, who are targeting instances which don’t have captcha enabled.

There are folks that are running their own instances as well, as single user instances or are working to get the federation to the point to open it up in anticipation for a larger flood. That doesnt make us spammers at all.

The questions of how to handle it are legitimate. In the end I feel the "fediverse" will need some user only instances (that is instances that just host users and not loads of communities) to help with the load and scaling issues MANY are seeing. Beehaw seems to have handled the influx to date the best, others like lemmy.ml and lemmy.world seem to have service level impacts that I can only really assume is due to scaling and load. And thats supposed to be the entire point right?

[-] Saik0Shinigami@lemmy.saik0.com 38 points 1 year ago

You ALL have a responsibility to communicate back to lemmy devs to try to stop it.

No I don't. Stop trying to brigade people to an issue. If you have an issue with it... Fork the lemmy UI code and make your own. Or stay on pre 0.18 code.

It's one thing to bring awareness to the issue. It's another to demand that I take action on something that's not only a non-issue for me (and likely many other admins of instances) but that the devs don't have to support. You're not paying them... you're not their mother. You don't get to force them to do anything they don't want to do.

Honestly the captchas that lemmy uses are terrible anyway. https://addons.mozilla.org/en-US/firefox/addon/2captcha-solver/ You can even solve them yourself as a browser extension... There's no point to them in today's world.

[-] speff@melly.0x-ia.moe 14 points 1 year ago

You’re not paying them… you’re not their mother. You don’t get to force them to do anything they don’t want to do.

I'm trying to think of what it would be like if one of my projects had a defined roadmap and then I suddenly get hundred of messages a day telling me I have to do something. lol, no. Maybe if I was actually being paid well for the project.

load more comments (14 replies)
[-] Steve@compuverse.uk 35 points 1 year ago* (last edited 1 year ago)

There are other options.

I'm just a hobbyist, but I have built a couple websites with a few hundred users.

A stupidly simple and effective option I've been using for several years now, is adding a dummy field to the application form. If you add an address field, and hide it with CSS, users won't see it and leave it blank. Bots on the other hand will see it and fill it in, because they always fill in everything. So any application that has an address can be automatically dropped. Or at least set aside for manual review.

I don't know how long such a simple trick will work on larger sites. But other options are possible.

[-] th3raid0r@tucson.social 8 points 1 year ago

Fun fact, I purposefully goaded the bots into attacking my instance.

Turns out they aren't even using the web form, they're going straight to the register api endpoint with python. The api endpoint lives at a different place from the signup page and putting a captcha in front of that page was useless in stopping the bots. Now, we can't just challenge requests going to the API endpoint since it's not an interactive session - it would break registration for normal users as well.

The in-built captcha was part of the API form in a way that prevented this attack where the standard Cloudflare rules are either too weak (providing no protection) or too strong (breaking functionality).

In my case I had to create some special rules to exclude python clients and other bots while making sure to keep valid browser attempts working. It was kind of a pain, actually. There's a lot of Lemmy that seems to trip the optional OWASP managed rules so there's a lot of "artisanally crafted" exclusions to keep the site functional.

Anyways, I guess my point is form interaction is just one way to spam sites, but this particular attacker is using the backend API and forgoing the sign-up page entirely. Hidden fields wouldn't be useful here, IMO.

[-] bionicjoey@lemmy.ca 8 points 1 year ago

Couldn't the bots just be programmed to not fill out that field? Or not fill out any field flagged as hidden?

[-] Steve@compuverse.uk 15 points 1 year ago

You'd think so.

But it's not flagged as hidden. Instead you use CSS to set display as none. So the bot needs to do more than look at the direct HTML. It needs to fully analyze all the linked HTML, CSS, and even JavaScript files. Basically it needs to be as complex as a whole browser. It can't be a simple script anymore. It becomes impracticality complicated for the not maker.

[-] Hyperz@beehaw.org 9 points 1 year ago

This might work against very generic bots, but it won't work against specialized bots. Those wouldn't even need to parse the DOM, just recreate the HTTP requests.

[-] Steve@compuverse.uk 9 points 1 year ago

Which is why you'd need something else for popular sites worth targeting directly. But there are more options than standard capta's. Replacing them isn't necessarily a bad idea.

load more comments (5 replies)
load more comments (1 replies)
load more comments (3 replies)
[-] Zetaphor@zemmy.cc 25 points 1 year ago

Nutomic has said they're open to restoring captchas, but it will require a fair amount of work to bring the 0.17 implementation into 0.18, which the currently don't have the bandwidth to implement.

They've also said they're open to PR's, so if someone really wants this feature they can open a PR for inclusion in the 0.18 release

NO it is not the best we can do, we need to be applying some pressure to the developers here and that requires EVERYONE to do their part.

I sure hope you're supporting them financially considering the demands you're making that require their time and labor.

[-] DrWeevilJammer@lm.rdbt.no 11 points 1 year ago

Someone has already submitted a PR with the changes the dev recommended. The captcha stuff is in a new db table instead of in-memory at the websocket server.

However, from one of the devs:

One note, is that captchas (and all signup blocking methods) being optional, it still won't prevent people from creating bot-only instances. The only effective way being to block them, or switch to allow-only federation.

Once people discover the lemmy-bots that have been made that can bypass the previous captcha method, it also won't help (unless a new captcha method like the suggested ones above are implemented).

The root of the issue seems to be that they've removed websockets, for the following reasons:

Huge burden to maintain, both on the server and in lemmy-ui. Possible memory leaks. Not scalable.

I can understand them wanting to make their lives a bit easier (see "huge burden to maintain) - Lemmy has exploded recently (see "not scalable") and there are far bigger issues to fix, and an even larger number of bad actors (see "possible memory leaks") who have learned about Lemmy at the same time as everyone else and want to exploit or break it.

[-] fragmentcity@lemm.ee 24 points 1 year ago

Despite what you're implying, the devs have no duty to fix admin-reported problems using admin-dictated solutions.

They have already said they would accept a PR adding support for captchas. Someone will undoubtedly do this before long.

Until then, why the urgency? What is it that's preventing you from keeping your instance on 0.17?

[-] th3raid0r@tucson.social 10 points 1 year ago

I disagree, once your open source project "sprouts wings" you enter an unspoken power battle. If enough of the community disagrees with something the chance of a successful fork grows. Once a project is forked away, you no longer have any control at all.

Also, even if I don't upgrade to v0.18, I have to live in a fediverse that have other instances that WILL, and they might pose a problem with increased spam.

[-] Dusty@lemmy.dustybeer.com 13 points 1 year ago* (last edited 1 year ago)

I disagree, once your open source project “sprouts wings” you enter an unspoken power battle

You've seen Hackers one too many times. Again you can run your instance however you want, and can defederate from instances that don't implement things they way you are demanding they should, but you do not dictate how others (or the developers) run things.

The beauty of open source is you can always fork your own. The beauty of federation is you can block whoever you want or whatever instance you want.

Other than that, you have no right to demand anything of anyone.

[-] th3raid0r@tucson.social 12 points 1 year ago

No, I was around when SysV Init was "replaced" by Systemd and how that impacted the Debian project (and other distros).

But you know what, sure, let's stick to your bad faith, insulting interpretation, after all it is more becoming of an internet troll. I'm sure it'll get you lots of updoots from similarly trollish individuals.

Personally, I believe in something called collective responsibility, and that does including expecting community members to do their fair share. But it sounds like you envision federations as mini fiefdoms.

load more comments (2 replies)
[-] fragmentcity@lemm.ee 9 points 1 year ago

Also, even if I don’t upgrade to v0.18, I have to live in a fediverse that have other instances that WILL, and they might pose a problem with increased spam.

A fork avoids this problem how?

I disagree, once your open source project “sprouts wings” you enter an unspoken power battle. If enough of the community disagrees with something the chance of a successful fork grows. Once a project is forked away, you no longer have any control at all.

Who's writing the code for the fork? If you see them, can you ask them to just submit the PR that the devs said they'll approve?

load more comments (3 replies)
[-] speff@melly.0x-ia.moe 8 points 1 year ago* (last edited 1 year ago)

... once again, the devs already said they would accept a PR with mCapchas. I don't see why any capable dev would fork a project rather than just contribute code. The community can disagree all they want - it takes actual programmers to split.

And if other instances start becoming spambots, just defederate.

load more comments (6 replies)
[-] Stumblinbear@pawb.social 22 points 1 year ago

Captchas pretty much worthless. They're easily bypassed for basically free. You're better off putting your instance behind Cloudflare with their captcha

[-] th3raid0r@tucson.social 11 points 1 year ago

Okay, so do you mind explaining why the servers onboarding the most spam users are the ones without Captchas?

If they are so ineffective, why are they effective now?

[-] Stumblinbear@pawb.social 12 points 1 year ago

Invisible captchas are about as useful as graphical ones and are significantly less annoying to the end user

[-] th3raid0r@tucson.social 13 points 1 year ago

Sure, so implement them in v.0.18 rather than leaving that essential feature for a future release - that's all I personally want.

I don't care about the technical implementation of the Captcha, but given the current threat landscape of low effort bot attacks, removing the feature in the meantime just makes the fediverse worse off.

load more comments (9 replies)
load more comments (7 replies)
load more comments (2 replies)
[-] Dusty@lemmy.dustybeer.com 20 points 1 year ago

I find myself really ticked off at other Admin’s abdication of duty when it comes to engaging with the developers.

Abdication of duty? Seriously? Do you think this is a job for people? Or that people that want a privacy related instance are "abdicating their duty" by not using captcha? Talk about hyperbole.

Run your instance how you want. Raise an issue with the devs if you want. Throw a fit if you want. But do not attempt to tell others how to run their instances or talk for other people and their "duties" when it comes to their own servers.

[-] xtremeownage@lemmyonline.com 20 points 1 year ago

We need to make it VERY clear that Captcha is required before v0.18’s release. Not after when we’ll all be scrambling…

You would honestly be surprised. Captcha isn't nearly as effective at stopping spam. It only stops the lowest hanging fruit.

Most of the "spambot" developers, started using AI-based tools a while back.

It only helps stopping the lowest-hanging of fruit.

Also, due to the way federation and all works.... well, just remember, there are a million ways for spammers to get access currently...

[-] gloriousspearfish@feddit.dk 9 points 1 year ago

Not just AI tools. They outsource captcha solving to cheap human labour.

load more comments (1 replies)
load more comments (2 replies)
[-] barsoap@lemm.ee 17 points 1 year ago* (last edited 1 year ago)

I know I'm veering kinda OT right now but speaking of captchas, they can also be used as a troll throttle by requiring captchas for posting if heuristics (think spamassassin) say that a user is being inflammatory, or falling for troll bait, or such. In case you understand German, have a video.

One specific feature of such a system is that it never absolutely denies users to post their comment as-is, but it may require them to solve multiple captchas (by claiming that the previous ones failed). That is, it bogs down to a simple psychological equation: Do I really care about being an assclown or feeding trolls enough to jump through those hoops. Especially the discouraging of troll feeding is highly effective as when trolls don't get engagement, they leave.

[-] reric88@beehaw.org 13 points 1 year ago

Glad to know I was here and did my part by reading this post. We couldn't have succeeded without me!🫡

[-] poVoq@slrpnk.net 13 points 1 year ago

Just enable admin approval and put a sensible registration rate limit. Works better without being a massive accessibility problem with dubious help against bots.

[-] th3raid0r@tucson.social 10 points 1 year ago

Sure, that might work for me, but it doesn't scale well for many other larger instances.

I'm not saying to not improve, quite the contrary, improvement is important. I'm saying don't take away the ONE thing that's preventing the spam issue from getting worse.

To be clear, I am a developer in real life. I'm not just talking out of my ass. There are way to roll out a new implementation without leaving everyone exposed.

[-] DrJenkem@lemmy.blugatch.tube 10 points 1 year ago

Since you're a dev, submit a PR for a new captcha. I'm not even using the feature on my instance as I have open signups disabled. So no, I won't be hassling the devs. If something comes up that I want changed badly enough, I'll implement it myself.

load more comments (1 replies)
[-] redcalcium@c.calciumlabs.com 12 points 1 year ago

Related issue: https://github.com/LemmyNet/lemmy/issues/3204

The devs seems to prefer mCaptcha (a proof-of-work captcha) than graphical captchas.

load more comments (3 replies)
[-] Suppoze@beehaw.org 11 points 1 year ago

Who is impacted? Everyone, it just instances upgrading to 0.18?

To be honest, your post doesn't really explain the current situation and impact It's a call to arms, but I have no idea how important it impactful it is.

[-] th3raid0r@tucson.social 9 points 1 year ago

Everyone is impacted, but especially moderators and admins. Moderators will see more spam if Capcha is removed, even if their own instance isn't on v0.18 - they will exist in a fediverse with instances that are on v0.18.

Admins are impacted because Captcha served as a decent way, when coupled with email validation, to combating spam account sign ups.

load more comments (1 replies)
[-] xtremeownage@lemmyonline.com 8 points 1 year ago

Hunh.

I just had a surge of user registrations on my instance.

All passed the captcha. All passed the email validation.

All, had a valid-sounding response.

I am curious to know if they are actual users, or.... if I just became the host of a spam instance. :-/

Doesn't appear to be an easy way to determine.

[-] th3raid0r@tucson.social 12 points 1 year ago

Hmmm, I'd check the following:

  1. Do the emails follow a pattern? (randouser####@commondomain.com)
  2. Did the emails actually validate, or do you just not see bouncebacks? There is a DB field for this that admins can query (i'll dig it up after I make this high level post)
  3. Did the surge come from the same IP? Multiple? Did it use something that doesn't look like a browser?
  4. Did the surge traffic hit /signup or did it hit /api/v3/register exclusively?

With those answers I should be able to tell if it's the same or similar attacker getting more sophisticated.

Some patterns I noticed in the attacks I've received:

  1. it's exactly 9 attempts every 30 minutes from the user agent "python/requests"
  2. The users that did not get an email bounceback were still not authenticated hours later (maybe the attacker lucked out with a real email that didn't bounce back?). There was no effort to verify from what I could determine.

Some vulnerabilities I know that can be exploited and would expect to see next:

  1. ChatGPT is human enough sounding for the registration forms. I've got no idea why folks think this is the end-all solution when it could be faked just as easily.
  2. Duplicate Email conflicts can be bypassed by using a "+category" in your email. ie (someuser+lemmy@somedomain.com) This would allow someone to associate potentially hundreds of spam accounts with a single email.
load more comments (16 replies)
load more comments (5 replies)

Just created the instance, now the spammers consumed all my email allowances today :( I just enabled CAPTCHA now, now I'm gonna wait till tomorrow how much this makes difference.

load more comments
view more: next ›
this post was submitted on 20 Jun 2023
211 points (100.0% liked)

Technology

37208 readers
162 users here now

Rumors, happenings, and innovations in the technology sphere. If it's technological news or discussion of technology, it probably belongs here.

Subcommunities on Beehaw:


This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.

founded 2 years ago
MODERATORS