Ok, something is busted with the lemmy API endpoint that shows current federation state. It is currently showing nearly all remote instances as dead:
But "dead" instances are still successfully receiving content from AZ, and sending back to us.
this post was submitted on 18 Jan 2024
0 points (NaN% liked)
Meta
623 readers
1 users here now
Discussion about the aussie.zone instance itself
founded 1 year ago
MODERATORS
Ok, there's more to this than I first thought. It seems there is a back end task set to run at a set time every day, if the instance is restarting at that time the task doesn't run... this task updates the instances table to show remote instances as "seen" by AZ. With the memory leaks in 0.19.1, the instance has been restarting when this task is running... leading to this situation.
I've updated the server restart cronjob to not run around the time this task runs... and I've again manually updated the DB to flag all known instances as alive rather than dead.
Will keep an eye on it some more...
For anyone curious, two of the bugs that are related to this:
https://github.com/LemmyNet/lemmy/issues/4288
https://github.com/LemmyNet/lemmy/issues/4039
Looks like what caused this might have been resolved? https://github.com/LemmyNet/lemmy/pull/4377
Excellent, hopefully that is included in 0.19.3 that came out overnight... will upgrade shortly.
It is. I found the page I linked from the release notes.
Cool, we're on 0.19.3 now.
How do things look after the past week?
0.19.3 still has the memory leak, I've left Lemmy to restart once a day to mitigate this. Federation seems to be fine for the last week after some minor changes and the upgrade.
I'm claiming federation issues as the reason my posts aren't getting thousands of upvotes.
I found that this post's body text is best viewed on desk/laptop for anyone trying to make heads or tails of all the links on mobile. Edit: Looks like my idea that this issue was due to some instances not being on 0.19.2 has been shot out of the water.