We've seen some issues with AZ again today. On investigating it looks like more DDOS/abuse traffic from overseas. I've applied the same sort of block as last time we saw this, and things appear to have returned to normal..
The sharp drop is when the "attack" started, the jump is when it was blocked... the red line is blocked requests.
Probably both this and cloudflare caching. Looks like I set CF to cache for 1 year at some point, I can lower it... but that won't "fix" this, only limit the time its an issue for.
Yes, and that has put a smile on my face.
His term has covered 4 months so far... Jan, Feb, Mar, Apr.
No idea where it's from, but I used it as a kid too. Also munted, though that has fallen out of use for the similar sound to another word.
You're wrong, I'll leave it at that. Won't be replying any further.
With the resources available its not feasible for AZ to develop/deploy custom solutions that can be resolved by remote instances with trivial configuration changes.
I'm not going to address specific parts of your post, suffice to say I disagree on almost everything you said.
As I said previously, if you have a workable solution please do devlop it and submit a PR to the lemmy devs. I'd be happy to try your suggestion should they roll it in.
You're contradicting yourself there. By definition adding an external service is a customization to lemmy. I'm not interested in running un-vetted software from a third party.
This has been discussed previously with a request from a reputable source to batching content from LW. That setup required an additional server for AZ, close to LW. And for LW to send their outgoing federation traffic for AZ to it, which then batched and send to the real AZ server. This offer was declined, though appreciated.
I've been transparent and open about the situation. You seem to think this is the fault of AZ, and we're willfully not taking an action that we should be taking. This is not the case.
As it stands the issue is inherent with single threaded lemmy federation, which is why the devs added the option for multiple concurrent threads. Until LW enable this feature, we'll see delayed content from them when their activity volume is greater than what can be federated with a single thread. To imply this is the fault of the receiving instances is disingenuous at best, and deliberately misleading at worst.
Note I said lemmy AND the activitypub protocol, ie lemmy does not currently have this capability. If it were added to mainline lemmy I'd be open to configuring it, but its not so I can't.
The root cause of the issue is well understood, the solution is available in lemmy already: multiple concurrent outgoing federation connections to remote instances. AZ has had this configured since it was available. LW have not yet enabled this, though they're now running a version that has it available.
Appreciate the offer, but I'm not interested in customising the AZ server configuration more than it already is. If you write it up and submit a PR that the main lemmy devs incorporate, I'd be happy to look at it.
That isn't how lemmy and the activitypub protocol work. Source instance pushes metadata about new content, remote instance then needs to pull it. If we've not received the push yet, we can't pull the additional info.
It sure doesn't feel like autumn so far
Not sure about the whole picture, but credit to Keating for bringing in compulsory superannuation.
Its been 6 months or so... figure its time for another of these. Keep in mind there have been some major config changes in the last week, which has resulted in the oddities below.
Graphs below cover 2 months, except Cloudflare which only goes to 30 days on free accounts.
CPU:
Memory:
Network:
Storage:
Cloudflare caching:
Comments: The server is still happily chugging along. Looking even happier now that I've properly migrated pict-rs to its integrated object storage config, rather than the bodged up setup.
RAM/CPU are all fine. Storage use is growing slowly as various databases grow. Still a long way from needing to purge old posts, if ever.
Cloudflare is saving less traffic these days, since Lemmy added support to proxy all images. Not a concern, well under the bandwidth cap for the server.
As usual feel free to ask any questions.
You've reminded me I should donate again, its been a while... though one small correction, O- is not the most common at all. It makes up under ~10% of the population as a rule: https://www.healthdirect.gov.au/blood-types