There was another attack going on (as you might have noticed). We're working on a fix. In the meantime, we've blocked the listing of comments, so we at least aren't down, but it did break comments.

Hope to have a fix in the next hour. Stay tuned!

Update OK we've implemented a fix, again many thanks to @[email protected] for his assistance. This will prevent the outages we've seen last couple of days. Let's see what they will come up with next...

A few days ago I saw some cool JoinLemmy stickers created by @[email protected] . I asked her if she could also create lemmy.world stickers, and she did!

You can see and order them here, also check the other cool stickers in her shop.

Thanks for creating them!

Update:
The comments from this post will not be removed as to preserve the discussion around the announcement. Any continued discussions outside of this thread that violate server rules will be removed. We feel that everyone that has an opinion, and wanted to vent, has been heard.

————-

Original post:
Yesterday, we received information about the planned federation by Hexbear. The announcement thread can be found here: https://www.hexbear.net/post/280770. After reviewing the thread and the comments, it became evident that allowing Hexbear to federate would violate our rules.

Our code of conduct and server rules can be found here.

The announcement included several concerning statements, as highlighted below:

• “Please try to keep the dirtbag lib-dunking to hexbear itself. Do not follow the Chapo Rules of Posting, instead try to engage utilizing informed rhetoric with sources to dismantle western propaganda. Posting the western atrocity propaganda and pig poop balls is hilarious but will pretty quickly get you banned and if enough of us do it defederated.”
• “The West's role in the world, through organizations such as NATO, the IMF, and the World Bank - among many others - are deeply harmful to the billions of people living both inside and outside of their imperial core.”
• “These organizations constitute the modern imperial order, with the United States at its heart - we are not fooled by the term "rules-based international order." It is in the Left's interest for these organizations to be demolished. When and how this will occur, and what precisely comes after, is the cause of great debate and discussion on this site, but it is necessary for a better world.”

The rhetoric and goal of Hexbar are clear based on their announcement: to "dismantle western propaganda" and "demolish organizations such as NATO” shows that Hexbar has no intention of "respecting the rules of the community instance in which they are posting/commenting.” It’s to push their beliefs and ideology.

In addition, several comments from a Hexbear admin, demonstrate that instance rules will not be respected.

Here are some examples:

“I can assure you there will be no lemmygrad brigades, that energy would be better funneled into the current war against liberalism on the wider fediverse.”

“All loyal, honest, active and upright Communists must unite to oppose the liberal tendencies shown by certain people among us, and set them on the right path. This is one of the tasks on our ideological front.”

• https://lemmy.world/comment/121850
• https://lemmy.world/comment/1487168
• https://lemmy.world/comment/1476084
• https://lemmy.world/comment/171595
• https://www.hexbear.net/comment/3648500

Overall community comments:

• https://www.hexbear.net/comment/3526128
• https://www.hexbear.net/comment/3526086
• https://www.hexbear.net/comment/3652828

To clarify, for those who have inquired about why Hexbear versus Lemmygrad, it should be noted that we are currently exploring the possibility of defederating from Lemmygrad as well based on similar comments Hexbear has made.

• https://lemmygrad.ml/post/158656
• https://lemmygrad.ml/comment/882559
• https://lemmygrad.ml/comment/540170
• https://lemmygrad.ml/comment/446529

Defederation should only be considered as a last resort. However, based on their comments and behavior, no positive outcomes can be expected.

We made the decision to preemptively defederate from Hexbear for these reasons. While we understand that not everyone may agree with our decision, we believe it is important to prioritize the best interests of our community.

Lemmy.world has been down between 02:00 UTC and 05:45 UTC. This was caused by the database spiking to 100% cpu (all 32 cores/64 threads!) due to inefficient queries been fired to the db very often.

I’ve collected the logs and we’ll be checking how to prevent this. (And what caused this)

Update The upgrade was done, DB migrations took around 5 minutes. We'll keep an eye out for (new) issues but for now it seems to be OK.

Original message We will upgrade lemmy.world to 0.18.3 today at 20:00 UTC+2 (Check what this isn in your timezone). Expect the site to be down for a few minutes. ""Edit"" I was warned it could be more than a few minutes. The database update might even take 30 minutes or longer.

Release notes for 0.18.3 can be found here: https://github.com/LemmyNet/lemmy/blob/main/RELEASES.md

(This is unrelated to the downtimes we experienced lately, those are caused by attacks that we're still looking into mitigating. Sorry for those)

We just added Alexandrite to the server, it's an alternative desktop UI for Lemmy created by Sheodox who worked tirelessly to make the necessary changes to we could host it ourselves here. So go to https://a.lemmy.world and have a look!

He continues to update it constantly, you can follow the development on his github page or in his community. If you like what you see and want to support him, why not buy him a coffee? :)

For those who don't have Lemmy World as their home instance and want to use Alexandrite, either ask your instance admins to add it or go to https://alexandrite.app!

Edit: I should probably have mentioned that Alexandrite is meant for desktop!

Today, like the past few days, we have had some downtime. Apparently some script kids are enjoying themselves by targeting our server (and others). Sorry for the inconvenience.

Most of these 'attacks' are targeted at the database, but some are more ddos-like and can be mitigated by using a CDN. Some other Lemmy servers are using Cloudflare, so we know that works. Therefore we have chosen Cloudflare as CDN / DDOS protection platform for now. We will look into other options, but we needed something to be implemented asap.

For the other attacks, we are using them to investigate and implement measures like rate limiting etc.

As requested by some users: 'old' style now accessible via https://old.lemmy.world

Code can be found here: https://github.com/rystaf/mlmym , created by Ryan (Is he here?) (Yes he appears to be! @[email protected] ! Thanks for this awesome front-end!)

We have received numerous reports from users about the closure of the c/android community. While we fully support the original community owners' decision to move to another instance, it will eventually be necessary to open up the community on Lemmy.world. The beauty of the fediverse is that multiple communities on the same subject can exist in different instances. However, if you can no longer moderate a community on Lemmy for any reason, it is important to pass it on to individuals who are willing and able to do so.

To ensure the best interests of our instance members, it is necessary to establish boundaries. Holding onto a community name cannot be a permanent arrangement. It's important to consider our users' ongoing interest in the community if they wish it to continue. While we acknowledge the objective of consolidating communities, current community members ultimately decide whether they wish to join the new community at lemdro.id.

To ensure a smooth transition, we will keep the community locked for another week, providing ample time to inform the active user base about the move to the new instance at https://lemmy.world/c/[email protected].

Thanks to @[email protected] for another release with awesome enhancements, see release notes here: https://lemmy.world/post/1558795

I blogged about what happened in June, and the financial overview.

It's always the small things you overlook...

The docker-compose.yml I copied from somewhere when setting up lemmy.world apparently was missing the external network for the pictrs container.. So pictrs was working, as long as it got the images via Lemmy. Getting the images via URL didn't work...

Looks like it's working now. Looks a whole lot better with all the images :-)

Edit For existing posts: Edit the post, then Save. (No need to change anything). This also fetches the image.

(Duplicate post :-) see https://lemmy.world/post/1375042)

The lemmy.world instance was just updated to version 0.18.2. The login issues that were being reported (for example, here) are now resolved.

For release notes: https://lemmy.world/post/1339018

Edit for those who still have issues logging in:

• When using a browser: clear cookies and cache
• When using an app: remove your lemmy.world account and add it again.

We've installed Voyager and it's reachable at https://m.lemmy.world, you can browse Lemmy, and login there (also if your account isn't on lemmy.world)

PS Thanks go out to @stux@[email protected] , he came up with the idea (see https://m.geddit.social).

While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.

As I am told, this was the issue:

• There is an vulnerability which was exploited
• Several people had their JWT cookies leaked, including at least one admin
• Attackers started changing site settings and posting fake announcements etc

Our mitigations:

• We removed the vulnerability
• Deleted all comments and private messages that contained the exploit
• Rotated JWT secret which invalidated all existing cookies

The vulnerability will be fixed by the Lemmy devs.

Details of the vulnerability are here

Many thanks for all that helped, and sorry for any inconvenience caused!

Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been 'stolen' and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).

For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.

There has been significant discussion in recent weeks regarding Meta/Threads. We would like to express our disappointment with the negative and threatening tone of some of these discussions. We kindly ask everyone to engage in civil discourse and remember that not everyone will share the same opinions, which is perfectly acceptable.

When considering whether or not to defederate from Threads, we're looking for a decision based on facts that prioritize your safety. We strive to remain neutral to make an informed choice.

First, there seem to be some misconceptions about how the Fediverse operates based on several posts. We’ve compiled some resource links to help explain the details and address any misunderstandings.

Fed Tips , Fediverse , ActivityPub

Initial Thoughts:

It seems unlikely that Meta will federate with Lemmy. When/if Meta adopts ActivityPub, it will likely affect Mastodon only rather than Lemmy, given Meta's focus on being a Twitter alternative at the moment.

Please note that we have a few months before Threads will even federate with Mastodon, so we have some time to make the right decision.

Factors to Consider:
Factors to consider if Meta federates with Lemmy:

Privacy - While it’s true that Meta's privacy settings for the app are excessive, it’s important to note that these settings only apply to users of the official Threads app and do not impact Lemmy users. It’s worth mentioning that Lemmy does not collect any personal data, and Meta has no means of accessing such data from this platform. In addition, when it comes to scraping data from your post/comments, Meta doesn’t need ActivityPub to do that. Anyone can read your profile and public posts as it is today.

Moderation - If a server hosts a substantial amount of harmful content without performing efficient and comprehensive moderation, it will create an excessive workload for our moderators. Currently, Meta is utilizing its existing Instagram moderation tools. Considering there were 95 million posts on the first day, this becomes worrisome, as it could potentially overwhelm us and serve as a sufficient reason for defederation.

Ads - It’s possible if Meta presents them as posts.

Promoting Posts - It’s possible with millions of users upvoting a post for it to trend.

Embrace, extend, and extinguish (EEE) - We don't think they can. If anyone can explain how they technically would, please let us know. Even if Meta forks Lemmy and gets rid of the original software, Lemmy will survive.

Instance Blocking - Unlike Mastodon, Lemmy does not provide a feature for individual users to block an instance (yet). This creates a dilemma where we must either defederate, disappointing those who desire interaction with Threads, or choose not to defederate, which will let down those who prefer no interaction with Threads.

Blocking Outgoing Federation - There is currently no tool available to block outgoing federation from lemmy.world to other instances. We can only block incoming federation. This means that if we choose to defederate with our current capabilities, Threads will still receive copies of lemmy.world posts. However, only users on Threads will be able to interact with them, while we would not be able to see their interactions. This situation is similar to the one with Beehaw at the moment. Consequently, it leads to significant fragmentation of content, which has real and serious implications.

Conclusion:
From the points discussed above, the possible lack of moderation alone justifies considering defederation from Threads. However, it remains to be seen how Meta will handle moderation on such a large scale. Additionally, the inability of individuals to block an instance means we have to do what is best for the community.

If you have any added points or remarks on the above, please send them to @[email protected].

We've updated Lemmy.world to Lemmy 0.18.1.

For the release notes, see https://lemmy.world/post/1139237

For those who find it interesting, enjoy!

Another day, another update.

More troubleshooting was done today. What did we do:

• Yesterday evening @phiresky@[email protected] did some SQL troubleshooting with some of the lemmy.world admins. After that, phiresky submitted some PRs to github.
• @[email protected] created a docker image containing 3PR's: Disable retry queue, Get follower Inbox Fix, Admin Index Fix
• We started using this image, and saw a big drop in CPU usage and disk load.
• We saw thousands of errors per minute in the nginx log for old clients trying to access the websockets (which were removed in 0.18), so we added a return 404 in nginx conf for /api/v3/ws.
• We updated lemmy-ui from RC7 to RC10 which fixed a lot, among which the issue with replying to DMs
• We found that the many 502-errors were caused by an issue in Lemmy/markdown-it.actix or whatever, causing nginx to temporarily mark an upstream to be dead. As a workaround we can either 1.) Only use 1 container or 2.) set ~~proxy_next_upstream timeout;~~ max_fails=5 in nginx.

Currently we're running with 1 lemmy container, so the 502-errors are completely gone so far, and because of the fixes in the Lemmy code everything seems to be running smooth. If needed we could spin up a second lemmy container using the ~~proxy_next_upstream timeout;~~ max_fails=5 workaround but for now it seems to hold with 1.

Thanks to @[email protected] , @[email protected] , @[email protected], @[email protected] , @[email protected] , @[email protected] for their help!

And not to forget, thanks to @[email protected] and @[email protected] for their continuing hard work on Lemmy!

And thank you all for your patience, we'll keep working on it!

Oh, and as bonus, an image (thanks Phiresky!) of the change in bandwidth after implementing the new Lemmy docker image with the PRs.

Edit So as soon as the US folks wake up (hi!) we seem to need the second Lemmy container for performance. So that's now started, and I noticed the proxy_next_upstream timeout setting didn't work (or I didn't set it properly) so I used max_fails=5 for each upstream, that does actually work.

Status update July 4th

Just wanted to let you know where we are with Lemmy.world.

Issues

As you might have noticed, things still won't work as desired.. we see several issues:

Performance

• Loading is mostly OK, but sometimes things take forever
• We (and you) see many 502 errors, resulting in empty pages etc.
• System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)

Bugs

• Replying to a DM doesn't seem to work. When hitting reply, you get a box with the original message which you can edit and save (which does nothing)
• 2FA seems to be a problem for many people. It doesn't always work as expected.

Troubleshooting

We have many people helping us, with (site) moderation, sysadmin, troubleshooting, advise etc. There currently are 25 people in our Discord, including admins of other servers. In the Sysadmin channel we are with 8 people. We do troubleshooting sessions with these, and sometimes others. One of the Lemmy devs, @[email protected] is also helping with current issues.

So, all is not yet running smoothly as we hoped, but with all this help we'll surely get there! Also thank you all for the donations, this helps giving the possibility to use the hardware and tools needed to keep Lemmy.world running!

Looks like it works.

Edit still see some performance issues. Needs more troubleshooting

Update: Registrations re-opened We encountered a bug where people could not log in, see https://github.com/LemmyNet/lemmy/issues/3422#issuecomment-1616112264 . As a workaround we opened registrations.

Thanks

First of all, I would like to thank the Lemmy.world team and the 2 admins of other servers @[email protected] and @[email protected] for their help! We did some thorough troubleshooting to get this working!

The upgrade

The upgrade itself isn't too hard. Create a backup, and then change the image names in the docker-compose.yml and restart.

But, like the first 2 tries, after a few minutes the site started getting slow until it stopped responding. Then the troubleshooting started.

The solutions

What I had noticed previously, is that the lemmy container could reach around 1500% CPU usage, above that the site got slow. Which is weird, because the server has 64 threads, so 6400% should be the max. So we tried what @[email protected] had suggested before: we created extra lemmy containers to spread the load. (And extra lemmy-ui containers). And used nginx to load balance between them.

Et voilà. That seems to work.

Also, as suggested by him, we start the lemmy containers with the scheduler disabled, and have 1 extra lemmy running with the scheduler enabled, unused for other stuff.

There will be room for improvement, and probably new bugs, but we're very happy lemmy.world is now at 0.18.1-rc. This fixes a lot of bugs.

We'll give the upgrade new try tomorrow. I've had some good input from admins of other instances, which are also gonna help troubleshoot during/after the upgrade.

Also there are newer RC versions with fixed issues.

Be aware that might we need to rollback again, posts posted between the upgrade and the rollback will be lost.

We see a huge rise in new user signups (duh.. it's July 1st) which also stresses the server. Let's hope the improvements in 0.18.1 will also help with that.

So I've been troubleshooting the federation issues with some other admins:

(Thanks for the help)

So what we see is that when there are many federation workers running at the same time, they get too slow, causing them to timeout and fail.

I had federation workers set to 200000. I've now lowered that to 8192, and set the activitypub logging to debugging to get queue stats. RUST_LOG="warn,lemmy_server=warn,lemmy_api=warn,lemmy_api_common=warn,lemmy_api_crud=warn,lemmy_apub=warn,lemmy_db_schema=warn,lemmy_db_views=warn,lemmy_db_views_actor=warn,lemmy_db_views_moderator=warn,lemmy_routes=warn,lemmy_utils=warn,lemmy_websocket=warn,activitypub_federation=debug"

Also, I saw that there were many workers retrying to servers that are unreachable. So, I've blocked some of these servers:

commallama.social,mayheminc.win,lemmy.name,lm.runnerd.net,frostbyrne.io,be-lemmy.org,lemmonade.marbledfennec.net,lemmy.sarcasticdeveloper.com,lemmy.kosapps.com,pawb.social,kbin.wageoffsite.com,lemmy.iswhereits.at,lemmy.easfrq.live,lemmy.friheter.com,lmy.rndmm.us,kbin.korgen.xyz

This gave good results, way less active workers, so less timeouts. (I see that above 3000 active workers, timeouts start).

(If you own one of these servers, let me know once it's back up, so I can un-block it)

Now it's after midnight so I'm going to bed. Surely more troubleshooting will follow tomorrow and in the weekend.

Please let me know if you see improvements, or have many issues still.

We're closing this thread. Everything that could be said has been said. Thank you

Original Post:
Today, we want to inform everyone that we have decided to defederate from https://exploding-heads.com/. We understand that defederating should always be a last resort, and individuals can certainly block communities. However, blocking alone does not prevent potential harm to vulnerable communities.

After carefully reviewing the instance, reported posts, and multiple comments from the community, we have concluded that exploding-heads is not adhering to the Lemmy or Citizen Code of Conduct. Therefore, we cannot, in good faith, continue to federate with an instance that consistently promotes hate, racism, and bullying.

Examples:
https://lemmy.world/post/577526 - Community Moderator Harassment
https://exploding-heads.com/post/92194 - Systemadmin Post
https://exploding-heads.com/post/90780 - Systemadmin Post
https://exploding-heads.com/post/91488 - Systemadmin Post
https://exploding-heads.com/post/93725 - Community Moderator Post

Again, deciding to defederate from an instance is not taken lightly. In the future, we will continue to review instances on a case-by-case bases.

As for our community, please refrain from posting or commenting with hateful words as well. Arguing back and calling people names is not the solution. The best course of action is to report the posts or comments violating our server rules.

Lemmy Code of Conduct
https://join-lemmy.org/docs/code_of_conduct.html
Citizen Code of Conduct https://github.com/stumpsyn/policies/blob/master/citizen_code_of_conduct.md

“We are committed to providing a friendly, safe, and welcoming environment for all, regardless of level of experience, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, nationality, or other similar characteristic.”