Today, like the past few days, we have had some downtime. Apparently some script kids are enjoying themselves by targeting our server (and others). Sorry for the inconvenience.

Most of these 'attacks' are targeted at the database, but some are more ddos-like and can be mitigated by using a CDN. Some other Lemmy servers are using Cloudflare, so we know that works. Therefore we have chosen Cloudflare as CDN / DDOS protection platform for now. We will look into other options, but we needed something to be implemented asap.

For the other attacks, we are using them to investigate and implement measures like rate limiting etc.

As requested by some users: 'old' style now accessible via https://old.lemmy.world

Code can be found here: https://github.com/rystaf/mlmym , created by Ryan (Is he here?) (Yes he appears to be! @[email protected] ! Thanks for this awesome front-end!)

Thanks to @[email protected] for another release with awesome enhancements, see release notes here: https://lemmy.world/post/1558795

I blogged about what happened in June, and the financial overview.

It's always the small things you overlook...

The docker-compose.yml I copied from somewhere when setting up lemmy.world apparently was missing the external network for the pictrs container.. So pictrs was working, as long as it got the images via Lemmy. Getting the images via URL didn't work...

Looks like it's working now. Looks a whole lot better with all the images :-)

Edit For existing posts: Edit the post, then Save. (No need to change anything). This also fetches the image.

(Duplicate post :-) see https://lemmy.world/post/1375042)

We've installed Voyager and it's reachable at https://m.lemmy.world, you can browse Lemmy, and login there (also if your account isn't on lemmy.world)

PS Thanks go out to @stux@[email protected] , he came up with the idea (see https://m.geddit.social).

While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.

As I am told, this was the issue:

• There is an vulnerability which was exploited
• Several people had their JWT cookies leaked, including at least one admin
• Attackers started changing site settings and posting fake announcements etc

Our mitigations:

• We removed the vulnerability
• Deleted all comments and private messages that contained the exploit
• Rotated JWT secret which invalidated all existing cookies

The vulnerability will be fixed by the Lemmy devs.

Details of the vulnerability are here

Many thanks for all that helped, and sorry for any inconvenience caused!

Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been 'stolen' and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).

For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.

We've updated Lemmy.world to Lemmy 0.18.1.

For the release notes, see https://lemmy.world/post/1139237

For those who find it interesting, enjoy!

Another day, another update.

More troubleshooting was done today. What did we do:

• Yesterday evening @phiresky@[email protected] did some SQL troubleshooting with some of the lemmy.world admins. After that, phiresky submitted some PRs to github.
• @[email protected] created a docker image containing 3PR's: Disable retry queue, Get follower Inbox Fix, Admin Index Fix
• We started using this image, and saw a big drop in CPU usage and disk load.
• We saw thousands of errors per minute in the nginx log for old clients trying to access the websockets (which were removed in 0.18), so we added a return 404 in nginx conf for /api/v3/ws.
• We updated lemmy-ui from RC7 to RC10 which fixed a lot, among which the issue with replying to DMs
• We found that the many 502-errors were caused by an issue in Lemmy/markdown-it.actix or whatever, causing nginx to temporarily mark an upstream to be dead. As a workaround we can either 1.) Only use 1 container or 2.) set ~~proxy_next_upstream timeout;~~ max_fails=5 in nginx.

Currently we're running with 1 lemmy container, so the 502-errors are completely gone so far, and because of the fixes in the Lemmy code everything seems to be running smooth. If needed we could spin up a second lemmy container using the ~~proxy_next_upstream timeout;~~ max_fails=5 workaround but for now it seems to hold with 1.

Thanks to @[email protected] , @[email protected] , @[email protected], @[email protected] , @[email protected] , @[email protected] for their help!

And not to forget, thanks to @[email protected] and @[email protected] for their continuing hard work on Lemmy!

And thank you all for your patience, we'll keep working on it!

Oh, and as bonus, an image (thanks Phiresky!) of the change in bandwidth after implementing the new Lemmy docker image with the PRs.

Edit So as soon as the US folks wake up (hi!) we seem to need the second Lemmy container for performance. So that's now started, and I noticed the proxy_next_upstream timeout setting didn't work (or I didn't set it properly) so I used max_fails=5 for each upstream, that does actually work.

Status update July 4th

Just wanted to let you know where we are with Lemmy.world.

Issues

As you might have noticed, things still won't work as desired.. we see several issues:

Performance

• Loading is mostly OK, but sometimes things take forever
• We (and you) see many 502 errors, resulting in empty pages etc.
• System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)

Bugs

• Replying to a DM doesn't seem to work. When hitting reply, you get a box with the original message which you can edit and save (which does nothing)
• 2FA seems to be a problem for many people. It doesn't always work as expected.

Troubleshooting

We have many people helping us, with (site) moderation, sysadmin, troubleshooting, advise etc. There currently are 25 people in our Discord, including admins of other servers. In the Sysadmin channel we are with 8 people. We do troubleshooting sessions with these, and sometimes others. One of the Lemmy devs, @[email protected] is also helping with current issues.

So, all is not yet running smoothly as we hoped, but with all this help we'll surely get there! Also thank you all for the donations, this helps giving the possibility to use the hardware and tools needed to keep Lemmy.world running!