Improving NodeBB for High Forum Traffic

[crazycells]

hi @baris , while we are on this topic, I have some questions for you.

On some special days, our forum gets very crowded and we have high traffic problems.

this was from May this year, and here is our forum log for that day (~20 hours of log)
https://app.box.com/s/unx9pkmamlv7rsyimis5irgvm2sanpa7
(it was too long and large for Pastebin, so I am sharing the text file)

the situation was way worse in the past, but thanks to your and @julian 's suggestions for nginx and extra ports, we are getting better every year at handling this, although we are still not there yet. Usually, we increase our CPU and RAM for this month, and on this specific day, we turn off "iframely" and "embed" plugins to prevent any extra loads to the forum. We keep getting "Internal Error" and "your connection was lost" problems the whole day. And I wonder what else we can do to improve this?

In the past we had memory and CPU problems, but this year we did not reach 100% levels although got very close, and additionally, I see lots of "2factor-plugin" errors in the log. So for next time, we are planning to turn off 2FA plugin and additionally increase RAM ve CPU since the levels were over 90%. What else can we do to improve?

Here are screenshots from the forum about the errors:

(online levels were fluctuating dramatically between 500-1000, I think this was because of lost connection errors)

(this internal error was quite common for this day)

(and even if we can open a page, in 10-20 seconds we would lose connection to the forum)

(this is how it looks in ACP )

[crazycells]

and here are some screenshots from Digital Ocean:

[baris]

At that many online users we usually scale to multiple droplets, you could investigate that.
Load balancer => 2 or more droplets to run nodebb => Database server that has enough ram to fit mongodb indexes in ram.
Other than that makes sure your nginx config is serving all static assets. You want nodebb processes to only server api requests. Logs can cause significant cpu usage if they are constantly logging useless info. Seems like 2 factor plugin is logging alot of these.

2022-05-07T16:00:04.433Z [4567/32790] - _[33mwarn_[39m: [plugins]
Callback timed out, hook 'static:sockets.validateSession' in plugin
'nodebb-plugin-2factor'

That hook is fired on every socket.emit so it might be overloading your backend/db. Are you running everything on a single droplet?

[julian]

Looking at those numbers, definitely.

We typically opt for a horizontally scaled solution the moment we see the "concurrent users" metric hit 100. This is a conservative estimate, NodeBB can easily handle more, as you've experienced, but it is a good point to start thinking about solutions.