295
submitted 11 months ago by Wander@yiffit.net to c/fediverse@lemmy.world

This new version introduced a system so that your instance stops sending out content to other instances that are supposedly dead / offline.

Unfortunately for some reason there's false positives. When I checked comparing the results from a curl request vs the information in our Lemmy database I found over 350+ false positives.

In the DB there is a table called "instance" which has a column called "updated". If the date on that column is older than 3 days, your server will stop sending any content to those instances.

For some reason I had entries that were dated as last being alive in July, while actually they were always up. If an entry is incorrect, you can fix it by manually using an update statement and adding today's date. If your instance is not too large you can safely update all entries to today's date and check if everything works as expected from then on any new content created on your instances.

The dead instances won't have an impact unless your instance is larger and generates more content, thus it might be easier to simply update all entries and have Lemmy believe they're all alive if you start noticing wonky behavior and don't want to check one by one.

If you don't know how to access the database run this command where domaincom is your instance domain without the dot.

  1. docker exec -it domaincom_postgres_1 busybox /bin/sh

  2. psql -U

(The default user is 'lemmy') You could technically do this is one single step, but it's good to know the command to get shell access to the container itself if you didn't know how to.

This should give you access to a postgres CLI interface. Use \c to connect, \dt to list tables and \d+ tablename to list table definition. You can also run SQL queries from there.

Try with this query: SELECT * from instance to list all instances and their updated date.

You can use other SQL queries to get better results or correct false positives. Just be careful with what you execute since there's no undo.

all 23 comments
sorted by: hot top controversial new old
[-] Shadow@lemmy.ca 37 points 11 months ago* (last edited 11 months ago)

One liner to get into postgres:

docker exec -it lemmyca_postgres_1 psql -U lemmy

Show only the instances not updated in past 3 days:

 select * from instance where updated < current_date-3;

Fix those old rows:

update instance set updated = current_date where updated < current_date - 3;
[-] DoomBot5@lemmy.world 4 points 11 months ago

Almost a 1 liner, just need to pass in the statement using the -c flag

[-] SeeJayEmm@lemmy.procrastinati.org 4 points 11 months ago

Your less than's got turned into <

[-] MarsAgainstVenus@fedimav.win 11 points 11 months ago* (last edited 11 months ago)

If you used Lemmy-Easy-Deploy, the docker name is lemmy-easy-deploy-postgres-1

docker exec -it lemmy-easy-deploy-postgres-1 busybox /bin/sh

[-] nix@merv.news 5 points 11 months ago

Very useful thanks for sharing! We should have an admin tips community to collect information like this

https://wiki.lemmyadmin.site

You can post it here and the explanation why it happens.

[-] Die4Ever@programming.dev 5 points 11 months ago

Is there a link to a GitHub issue or pull request that I can track for this?

[-] Xanvial@lemmy.world 4 points 11 months ago

Is it possible to add some scheduled trigger to check all instances in that table regardless of the last updated value? Shouldn't too frequent but enough to avoid fully defederating some instances

[-] Wander@yiffit.net 4 points 11 months ago* (last edited 11 months ago)

In theory that's what Lemmy now does every day, but I have no idea why it fails to update some instances sometimes. Instances which are very much alive at 12AM which is when this gets executed.

[-] redcalcium@lemmy.institute 9 points 11 months ago* (last edited 11 months ago)

If you're unlucky enough to have lemmy.world momentarily down at the exact same moment your instance doing the check, then it's possible. Also, lemmy.world implements rate limiting via cloudflare, and their admins probably not whitelisting requests from the dead instance checker.

I think instance owners that use cloudflare should whitelist /nodeinfo/2.0.json so automated requests to that endpoint is not blocked by cloudflare's bot protection (which can lead to failed daily instance check on instances you federated with).

[-] Wander@yiffit.net 4 points 11 months ago

Good point. I think that might be it actually. This could be the reason.

[-] theterrasque@infosec.pub 2 points 11 months ago

I hope that's not 12am on the dot for all instances, with no fudging of the check time.

[-] Wander@yiffit.net 1 points 11 months ago

That seems to be the case more or less. In my case times go from 0:00 to 0:25 or so when it finishes.

[-] theterrasque@infosec.pub 6 points 11 months ago

So at that exact time big servers get a few hundred requests? I hope it's a very light weight check and doesn't trigger any flood or spam protection.. otherwise https://i.imgflip.com/xhss9.jpg

[-] 1984 4 points 11 months ago

Thank you but is this really safe?

Just asking since there was db errors that messed up other people's instances, or so it seemed.

[-] Wander@yiffit.net 5 points 11 months ago

It should be safe as long as you put in a valid timestamp and not some other value. If you run a large instance, then you run the risk of pseudo-DDoSing yourself by sending a large amount of requests to dead servers, but unless you're a large instance you shouldn't have to worry about that.

[-] SeeJayEmm@lemmy.procrastinati.org 3 points 11 months ago

It's effectively how lemmy pre-0.18.3 behaved so it'd be no worse than that.

[-] soeren@iusearchlinux.fyi 1 points 11 months ago

thank you :)

this post was submitted on 04 Aug 2023
295 points (99.3% liked)

Fediverse

26737 readers
15 users here now

A community to talk about the Fediverse and all it's related services using ActivityPub (Mastodon, Lemmy, KBin, etc).

If you wanted to get help with moderating your own community then head over to !moderators@lemmy.world!

Rules

Learn more at these websites: Join The Fediverse Wiki, Fediverse.info, Wikipedia Page, The Federation Info (Stats), FediDB (Stats), Sub Rehab (Reddit Migration), Search Lemmy

founded 1 year ago
MODERATORS