this post was submitted on 04 Aug 2023
295 points (99.3% liked)

Fediverse

27952 readers
218 users here now

A community to talk about the Fediverse and all it's related services using ActivityPub (Mastodon, Lemmy, KBin, etc).

If you wanted to get help with moderating your own community then head over to !moderators@lemmy.world!

Rules

Learn more at these websites: Join The Fediverse Wiki, Fediverse.info, Wikipedia Page, The Federation Info (Stats), FediDB (Stats), Sub Rehab (Reddit Migration), Search Lemmy

founded 1 year ago
MODERATORS
 

This new version introduced a system so that your instance stops sending out content to other instances that are supposedly dead / offline.

Unfortunately for some reason there's false positives. When I checked comparing the results from a curl request vs the information in our Lemmy database I found over 350+ false positives.

In the DB there is a table called "instance" which has a column called "updated". If the date on that column is older than 3 days, your server will stop sending any content to those instances.

For some reason I had entries that were dated as last being alive in July, while actually they were always up. If an entry is incorrect, you can fix it by manually using an update statement and adding today's date. If your instance is not too large you can safely update all entries to today's date and check if everything works as expected from then on any new content created on your instances.

The dead instances won't have an impact unless your instance is larger and generates more content, thus it might be easier to simply update all entries and have Lemmy believe they're all alive if you start noticing wonky behavior and don't want to check one by one.

If you don't know how to access the database run this command where domaincom is your instance domain without the dot.

  1. docker exec -it domaincom_postgres_1 busybox /bin/sh

  2. psql -U

(The default user is 'lemmy') You could technically do this is one single step, but it's good to know the command to get shell access to the container itself if you didn't know how to.

This should give you access to a postgres CLI interface. Use \c to connect, \dt to list tables and \d+ tablename to list table definition. You can also run SQL queries from there.

Try with this query: SELECT * from instance to list all instances and their updated date.

You can use other SQL queries to get better results or correct false positives. Just be careful with what you execute since there's no undo.

all 23 comments
sorted by: hot top controversial new old
[–] Shadow@lemmy.ca 37 points 1 year ago* (last edited 1 year ago) (3 children)

One liner to get into postgres:

docker exec -it lemmyca_postgres_1 psql -U lemmy

Show only the instances not updated in past 3 days:

 select * from instance where updated < current_date-3;

Fix those old rows:

update instance set updated = current_date where updated < current_date - 3;
[–] DoomBot5@lemmy.world 4 points 1 year ago

Almost a 1 liner, just need to pass in the statement using the -c flag

Your less than's got turned into <

[–] MarsAgainstVenus@fedimav.win 11 points 1 year ago* (last edited 1 year ago)

If you used Lemmy-Easy-Deploy, the docker name is lemmy-easy-deploy-postgres-1

docker exec -it lemmy-easy-deploy-postgres-1 busybox /bin/sh

[–] nix@merv.news 5 points 1 year ago (2 children)

Very useful thanks for sharing! We should have an admin tips community to collect information like this

https://wiki.lemmyadmin.site

You can post it here and the explanation why it happens.

[–] Die4Ever@programming.dev 5 points 1 year ago

Is there a link to a GitHub issue or pull request that I can track for this?

[–] Xanvial@lemmy.world 4 points 1 year ago (1 children)

Is it possible to add some scheduled trigger to check all instances in that table regardless of the last updated value? Shouldn't too frequent but enough to avoid fully defederating some instances

[–] Wander@yiffit.net 4 points 1 year ago* (last edited 1 year ago) (2 children)

In theory that's what Lemmy now does every day, but I have no idea why it fails to update some instances sometimes. Instances which are very much alive at 12AM which is when this gets executed.

[–] redcalcium@lemmy.institute 9 points 1 year ago* (last edited 1 year ago) (1 children)

If you're unlucky enough to have lemmy.world momentarily down at the exact same moment your instance doing the check, then it's possible. Also, lemmy.world implements rate limiting via cloudflare, and their admins probably not whitelisting requests from the dead instance checker.

I think instance owners that use cloudflare should whitelist /nodeinfo/2.0.json so automated requests to that endpoint is not blocked by cloudflare's bot protection (which can lead to failed daily instance check on instances you federated with).

[–] Wander@yiffit.net 4 points 1 year ago

Good point. I think that might be it actually. This could be the reason.

[–] theterrasque@infosec.pub 2 points 1 year ago (1 children)

I hope that's not 12am on the dot for all instances, with no fudging of the check time.

[–] Wander@yiffit.net 1 points 1 year ago (1 children)

That seems to be the case more or less. In my case times go from 0:00 to 0:25 or so when it finishes.

[–] theterrasque@infosec.pub 6 points 1 year ago

So at that exact time big servers get a few hundred requests? I hope it's a very light weight check and doesn't trigger any flood or spam protection.. otherwise https://i.imgflip.com/xhss9.jpg

[–] 1984 4 points 1 year ago (1 children)

Thank you but is this really safe?

Just asking since there was db errors that messed up other people's instances, or so it seemed.

[–] Wander@yiffit.net 5 points 1 year ago (1 children)

It should be safe as long as you put in a valid timestamp and not some other value. If you run a large instance, then you run the risk of pseudo-DDoSing yourself by sending a large amount of requests to dead servers, but unless you're a large instance you shouldn't have to worry about that.

It's effectively how lemmy pre-0.18.3 behaved so it'd be no worse than that.

[–] soeren@iusearchlinux.fyi 1 points 1 year ago

thank you :)