this post was submitted on 04 Jun 2025

332 points (100.0% liked)

Fediverse

34222 readers

266 users here now

A community to talk about the Fediverse and all it's related services using ActivityPub (Mastodon, Lemmy, KBin, etc).

If you wanted to get help with moderating your own community then head over to [email protected]!

Rules

Posts must be on topic.
Be respectful of others.
Cite the sources used for graphs and other statistics.
Follow the general Lemmy.world rules.

Learn more at these websites: Join The Fediverse Wiki, Fediverse.info, Wikipedia Page, The Federation Info (Stats), FediDB (Stats), Sub Rehab (Reddit Migration)

founded 2 years ago

MODERATORS

[email protected]

332

Slrpnk.net outage (feddit.org)

submitted 4 days ago* (last edited 3 days ago) by [email protected] to c/[email protected]

123 comments fedilink hide all child comments

Alt account of [email protected] here.

Our instance is currently down and I can't get remote access to the servers. It appears that there might have been a hardware failure of the main firewall, which is the one thing I can't work around remotely.

I am still trying a few things, but I am not very optimistic that I can get access.

The really unfortunate part is that just now I am on one of my rare work deployments abroad, so I also can't access it physically during the next few weeks and my usual back up that could restart it is not available either.

As something like that never happened in 3 years operating the servers, I thought I can risk it, but murphy's law seems inescapable 😓

I will try to keep you posted here on any updates, but probably there will not be much I can do for a while. Really bad timing 😥

Edit: we might use this "opportunity" to migrate the instance to Piefed, which has been an idea for quite some time now. I will keep you posted on that.

top 50 comments

sorted by: hot top controversial new old

[–] [email protected] 1 points 14 hours ago* (last edited 14 hours ago) (1 children)

I am a slrpnk user. Oh well, shit happens. What makes me a bit sad was that a few months (?) ago something similar happened. Slrpnk.net was down for aa day or two? Don't remember the exact amount of time. What I do remember is asking if there was intention of adding a second admin and making the point that it would be good to create that sort of redundancy. I was ignored.

Mid of july is basically 1 and a half months from now. Hopefully a new solarpunk instance emerges, one that is safer from such scenarios.

Enjoy your vacation and don't stress too much! Shit happens.

[–] [email protected] 5 points 13 hours ago

I am genuinely confused. What makes you think you were ignored? We do have additional admins, the problem is physical access to the servers to replace hardware that seems to have broken.

We plan to add additional safeguards to such scenarios, but in the end it is a trade-off between reliability and costs. Running a Lemmy instance of our size on a fully managed rented server would easily triple the monthly costs. This would be beyond what I am willing to cover out of my own private pocket, thus making the instance dependent on donations. And that IMHO makes it more likely to be shut down in the future as donations are very variable.

I am not on vacations, but on a work deployment abroad, which further limits the time I can spend on this. Not that it would have made much of a difference, but at least the domain redirection and placeholder text was only possible to do on the weekend.

[–] [email protected] 4 points 1 day ago (1 children)

Thanks for letting us know what is going on.

[–] [email protected] 5 points 18 hours ago (1 children)

Sorry that it took so long to get the word out. I guess we relied a bit too much on the xmpp server for internal communication.

[–] [email protected] 4 points 7 hours ago

Even the big operations break in unexpected ways. Facebook accidentally broke the authentication for their door locks when they took the site down a few years back and had to fly people to a datacenter and break down a door to get to servers and restart things.

[–] [email protected] 5 points 2 days ago

I appreciate the update was thoroughly confused

[–] [email protected] 10 points 2 days ago (2 children)

Thanks for all that you do for us and this community! A little downtime won't hurt anyone. I would also be in gavour of PieFed migration 💙

[–] [email protected] 6 points 1 day ago (1 children)

I totally agree with everything you said :)

I just created an account in PieFed and it looks very interesting because there is the option to create your own feeds! I had the wrong impression that you can only choose from the existing ones. This is so exciting!!!

[–] [email protected] 4 points 1 day ago (1 children)

That sounds pretty cool. I don't know much about Piefed — would we still be able to access posts from Lemmy instances?

[–] [email protected] 5 points 1 day ago

Yes, they're 100% cross compatible. As an example, you can access [email protected] from your Lemmy instance seamlessly.

[–] [email protected] 6 points 2 days ago

I'd probably make an account there if it moves to piefed.

[–] [email protected] 21 points 3 days ago (1 children)

I this it’s a law that servers run 100% perfect until the literal day one leaves town with zero way to return home. One of the many reasons I got all my services off of unraid.

Very cool to learn you’re running your own machines. Do you go into detail about this anywhere?

[–] [email protected] 15 points 2 days ago* (last edited 2 days ago)

We have a small write up about the hardware on our wiki, but it is also down right now.

I think we will share a post-mortem write up of the actual improvements we will do to avoid this in the future.

One thing I will definitly do is to add a KVM remote management console to one of our server boards and move the main firewall into a VM with hardware passthrough of the NICs (this was anyways planned for a 10gbit network upgrade for the second half of 2025). This way I should be able to reboot and even reinstall the main ingress point remotely, so that only the fiber gateway remains as a failure point that requires physical access.

[–] [email protected] 1 points 3 days ago

Opengear cellular backup appliance with serial/console connectivity to hardware.

[–] [email protected] 8 points 3 days ago

Thanks for the heads up! My Main account is in slrpnk, so I'll set a reminder in an month to switch back. :)

[–] [email protected] 15 points 3 days ago

Thanks for the update, good luck with your trip.

[–] [email protected] 10 points 3 days ago

Thanks for letting us know! Try not to worry about it too much, it’s all good

[–] [email protected] 34 points 3 days ago (1 children)

Try not to stress too much about it while you're away! If nothing can be done, nothing can be done. Solarpunks need to be resilient to things like this anyway. Community over convenience.

[–] [email protected] 18 points 3 days ago

Thanks, I am trying, but it really is quite annoying to lose access like that.

load more comments