Totally healthy servers have a right to rest every once in a while too. Thanks for keeping us notified!
this post was submitted on 18 Jul 2024
202 points (97.6% liked)
Meta (lemm.ee)
3570 readers
4 users here now
lemm.ee Meta
This is a community for discussion about this particular Lemmy instance.
News and updates about lemm.ee will be posted here, so if that's something that interests you, make sure to subscribe!
Rules:
- Support requests belong in !support
- Only posts about topics directly related to lemm.ee are allowed
- If you don't have anything constructive to add, then do not post/comment here. Low effort memes, trolling, etc is not allowed.
- If you are from another instance, you may participate in discussions, but remain respectful. Realize that your comments will inevitably be associated with your instance by many lemm.ee users.
If you're a Discord user, you can also join our Discord server: https://discord.gg/XM9nZwUn9K
Discord is only a back-up channel, [email protected] will always be the main place for lemm.ee communications.
If you need help with anything, please post in !support instead.
founded 1 year ago
MODERATORS
Thank you for keeping us abreast of what's happening. I appreciate you, and how you manage this instance.
Seriously, your professionalism in handling the situation and in reporting it is fantastic.
It's totally above and beyond anything we should expect for a service powered by donations!
Thank you!
love you guys!
Sometimes, downtimes are awesome. Get off your machine and spend time with your family, folks!
All is forgiven, thank you for running this lovely instance ^_^
Would it be in bad taste to blame Russia?
Yeah, but it could have been China, India, Iran, or maybe even North Korea. There are a lot of places that think disrupting the rest of the world will get them somewhere.
I survived the July 18th lemm.ee downtime, and all I got was this lousy comment.
Typically when this happens, the issue is on the LB itself. Maybe its own network had issues?
I'd like to speak to a manager /s
Thanks for the quick fix! What did you have to do to get the load balancer working again?
For now, I just redeployed all of our servers completely, but as I don't know the actual root cause of the issue yet, I'm still investigating to figure out if anything more is needed.
Thanks for your great work and transperancy!
I thought the entire lemmy network was down because status.lemm.ee was saying our instance was fine and federation wasn't working with every other instance. lol
Is there another instance where you could report issues?
If we logged into another account, we’d be able to see those before it comes back up.
There are two useful sections on https://status.lemm.ee for this - firstly, there is an automated check for federation with all other instances on the bottom of the page, and everything there being red is a definite sign that something is wrong with lemm.ee itself. Secondly, near the top of that page, I will always write a status message manually when I discover & start work on any issues. This second part can have a bit of a delay, as it requires manual input from myself, but I have updated it every time we had any issues so far.
That’s good info. Thanks.
There's a Discord server in the sidebar that updates are posted in: https://discord.gg/XM9nZwUn9K
I know. I just don’t want to join a discord.
o7
Nginx? I had an nginx LB shit itself yesterday. Luckily it auto-recovered and I had HA but just weird it happened.
Actually, we're using Hetzner's cloud load balancer for lemm.ee. But if this issue repeats in the near future, then I will definitely consider setting up something else.
haproxy is where it's at!
It's probably a managed haproxy in Hetzner's case.
Thank goodness! Hopefully discovering these vulnerabilities and protecting them will help keep Lemmy alive when the big dogs come in to sweep us away! (Worst fears)
We appreciate what you do hero
I was wondering what was going on, status.lemm.ee said the server was ok but the federation was broken. Thank you for fixing it
Sorry for the delay in updating the status page - I actually had gone out for lunch just a few minutes before the downtime started, so I didn't even realize anything was up until I was back at my computer about 45 minutes later 💀
no need to apologise. still a better response time, than some of the professionals I work with ;-)