Looks like it works.

Edit still see some performance issues. Needs more troubleshooting

Update: Registrations re-opened We encountered a bug where people could not log in, see https://github.com/LemmyNet/lemmy/issues/3422#issuecomment-1616112264 . As a workaround we opened registrations.

Thanks

First of all, I would like to thank the Lemmy.world team and the 2 admins of other servers @[email protected] and @[email protected] for their help! We did some thorough troubleshooting to get this working!

The upgrade

The upgrade itself isn't too hard. Create a backup, and then change the image names in the docker-compose.yml and restart.

But, like the first 2 tries, after a few minutes the site started getting slow until it stopped responding. Then the troubleshooting started.

The solutions

What I had noticed previously, is that the lemmy container could reach around 1500% CPU usage, above that the site got slow. Which is weird, because the server has 64 threads, so 6400% should be the max. So we tried what @[email protected] had suggested before: we created extra lemmy containers to spread the load. (And extra lemmy-ui containers). And used nginx to load balance between them.

Et voilà. That seems to work.

Also, as suggested by him, we start the lemmy containers with the scheduler disabled, and have 1 extra lemmy running with the scheduler enabled, unused for other stuff.

There will be room for improvement, and probably new bugs, but we're very happy lemmy.world is now at 0.18.1-rc. This fixes a lot of bugs.

We'll give the upgrade new try tomorrow. I've had some good input from admins of other instances, which are also gonna help troubleshoot during/after the upgrade.

Also there are newer RC versions with fixed issues.

Be aware that might we need to rollback again, posts posted between the upgrade and the rollback will be lost.

We see a huge rise in new user signups (duh.. it's July 1st) which also stresses the server. Let's hope the improvements in 0.18.1 will also help with that.

So I've been troubleshooting the federation issues with some other admins:

(Thanks for the help)

So what we see is that when there are many federation workers running at the same time, they get too slow, causing them to timeout and fail.

I had federation workers set to 200000. I've now lowered that to 8192, and set the activitypub logging to debugging to get queue stats. RUST_LOG="warn,lemmy_server=warn,lemmy_api=warn,lemmy_api_common=warn,lemmy_api_crud=warn,lemmy_apub=warn,lemmy_db_schema=warn,lemmy_db_views=warn,lemmy_db_views_actor=warn,lemmy_db_views_moderator=warn,lemmy_routes=warn,lemmy_utils=warn,lemmy_websocket=warn,activitypub_federation=debug"

Also, I saw that there were many workers retrying to servers that are unreachable. So, I've blocked some of these servers:

commallama.social,mayheminc.win,lemmy.name,lm.runnerd.net,frostbyrne.io,be-lemmy.org,lemmonade.marbledfennec.net,lemmy.sarcasticdeveloper.com,lemmy.kosapps.com,pawb.social,kbin.wageoffsite.com,lemmy.iswhereits.at,lemmy.easfrq.live,lemmy.friheter.com,lmy.rndmm.us,kbin.korgen.xyz

This gave good results, way less active workers, so less timeouts. (I see that above 3000 active workers, timeouts start).

(If you own one of these servers, let me know once it's back up, so I can un-block it)

Now it's after midnight so I'm going to bed. Surely more troubleshooting will follow tomorrow and in the weekend.

Please let me know if you see improvements, or have many issues still.

We've upgraded lemmy.world to 0.18.1-rc.1 and rolled back that upgrade because of issues.

(If you had posted anything in those 10 minutes between upgrade and rollback, that post is gone. Sorry!)

The main issue we saw is that users can't login anymore. Existing sessions still worked, but new logins failed (from macos, ios and android. From linux and windows it worked)

Also new account creation didn't work.

I'll create an issue for the devs and retry once it's fixed.

Edit Contacted the devs, they tell me to try again with lemmy-ui at version 0.18.0. Will try again, brace for some downtime!

Edit 2 So we upgraded again, and it seemed to work nicely! But then it slowed down so much it was unuseable. There were many locks in the database. People reported many JSON errors. Sorry, we won't be on 0.18.1 any time soon I'm afraid..

I enabled the ipv6 address for lemmy.world. Should work now.

Next step would be enable dnssec, have to figure out how that worked again.

We're still working to find a solution for the posting slowness in large communities.

We have seen that a post does get submitted right away, but yet the page keeps 'spinning'

So right after you clicked 'Post' or 'Reply' you can refresh the page and the post should be there.

(But maybe to be sure you could copy the contents of your post first, so you can paste again if anything would go wrong..)

So after we've extended the virtual cloud server twice, we're at the max for the current configuration. And with this crazy growth (almost 12k users!!) even now the server is more and more reaching capacity.

Therefore I decided to order a dedicated server. Same one as used for mastodon.world.

So the bad news... we will need some downtime. Hopefully, not too much. I will prepare the new server, copy (rsync) stuff over, stop Lemmy, do last rsync and change the DNS. If all goes well it would take maybe 10 minutes downtime, 30 at most. (With mastodon.world it took 20 minutes, mainly because of a typo :-) )

For those who would like to donate, to cover server costs, you can do so at our OpenCollective or Patreon

Thanks!

Update The server was migrated. It took around 4 minutes downtime. For those who asked, it now uses a dedicated server with a AMD EPYC 7502P 32 Cores "Rome" CPU and 128GB RAM. Should be enough for now.

I will be tuning the database a bit, so that should give some extra seconds of downtime, but just refresh and it's back. After that I'll investigate further to the cause of the slow posting. Thanks @[email protected] for assisting with that.

(I'm creating a starting guide post here. Have patience, it will take some time...)

Disclaimer: I am new to Lemmy like most of you. Still finding my way. If you see something that isn't right, let me know. Also additions, please comment!

Welcome!

Welcome to Lemmy (on whichever server you're reading this)

About Lemmy

Lemmy is a federated platform for news aggregagtion / discussion. It's being developed by the Lemmy devs: https://github.com/LemmyNet

About Federation

What does this federation mean?

It means Lemmy is using a protocol (Activitypub) which makes it possible for all Lemmy servers to interact.

• You can search and view communities on remote servers from here
• You can create posts in remote communities
• You can respond to remote posts
• You will be notified (if you wish) of comments on your remote posts
• You can follow Lemmy users/communities on other platforms that also use Activitypub (like Mastodon, Calckey etc) (There's currently a known issue with that, see here

Please note that a server only starts indexing a server/community once it has been interacted with by a user of this server.

A great image describing this, made by @[email protected] : https://imgur.com/a/uyoYySY

About Lemmy.world

Lemmy.world is one of the many servers hosting the Lemmy software. It was started on June 1st, 2023 by @[email protected] , who is also running https://mastodon.world, https://calckey.world and others.

A list of Lemmy servers and their statistics can be found at FediDB

Quick start guide

Account

You can use your account you created to log in to the server on which you created it. Not on other servers. Content is federated to other servers, users/accounts are not.

Searching

In the top menu, you'll see the search icon. There, you can search for posts, communities etc.

You can just enter a search-word and it will find the Post-titles, post-content, communities etc containing that word that the server knows of. So any content any user of this server ever interacted with.

You can also search for a community by it's link, e.g. [[email protected]](/c/[email protected]). Even if the server hasn't ever seen that community, it will look it up remotely. Sometimes it takes some time for it to fetch the info (and displays 'No results' meanwhile..) so just be patient and search a second time after a few seconds.

Creating communities

First, make sure the community doesn't already exist. Use search (see above). Also try https://browse.feddit.de/ to see if there are remote communities on other Lemmy instances that aren't known to Lemmy.world yet.

If you're sure it doesn't exist yet, go to the homepage and click 'Create a Community'.

It will open up the following page:

Here you can fill out:

• Name: should be all lowercase letters. This will be the /c/
• Display name: As to be expected, this will be the displayed name.
• You can upload an icon and banner image. Looks pretty.
• The sidebar should contain things like description, rules, links etc. You can use Markdown (yey!)
• If the community will contain mainly NSFW content, check the NSFW mark. NSFW is allowed as long as it doesn't break the rules
• If you only want moderators to be able to post, check that checkbox.
• Select any language you want people to be able to post in. Apparently you shouldn't de-select 'Undetermined'. I was told some apps use 'Undetermined' as default language so don't work if you don't have it selected

Reading

I think the reading is obvious. Just click the post and you can read it. SOmetimes when there are many comments, they will partly be collapsed.

Posting

When viewing a community, you can create a new post in it. First of all make sure to check the community's rules, probably stated in the sidebar.

In the Create Post page these are the fields:

• URL: Here you can paste a link which will be shown at the top of the post. Also the thumbnail of the post will link there. Alternatively you can upload an image using the image icon to the right of the field. That image will also be displayed as thumbnail for the post.
• Title: The title of the post.
• Body: Here you can type your post. You can use Markdown if you want.
• Community: select the community where you want this post created, defaults to the community you were in when you clicked 'create post'
• NSFW: Select this if you post any NSFW material, this blurs the thumbnail and displays 'NSFW' behind the post title.
• Language: Specify in which language your post is.

Also see the Lemmy documentation on formatting etc.

Commenting

Moderating / Reporting

Client apps

There are some apps available or in testing. See this post for a list!

Issues

When you find any issue, please report so here: https://lemmy.world/post/15786 if you think it's server related (or not sure).

Report any issues or improvement requests for the Lemmy software itself here: https://github.com/LemmyNet

Known issues

Known issues can be found in the beforementioned post, one of the most annoying ones is the fact that post/reply in a somewhat larger community can take up to 10 seconds. It seems like that's related to the number of subscribers of the community.

I'll be looking into that one, and hope the devs are too.

Trying out Lemmy. It seems to work, e-mail and federation fixed now