Performance insights of NodeBB

Tomáš Nesrovnal

Hi,

We are experiencing performance issues with our NodeBB installation. It got significantly worse after upgrading to v3.x. The forum is noticeably slowing down at times - both loading and saving up posts is slow and often we get just a 502 response.

There are several factors that might cause this: overloaded database, attackers doing a brute force requests ( I created a issue about this here https://github.com/NodeBB/NodeBB/issues/12432 ), our api calls from various places.

I would appreciate any guidance or advice on how to at least diagnose these issues - what is the biggest bottleneck.

There are tools like ip blacklists, but can you recommend any tool that handles this automatically based on incoming requests? Like a IP doing many calls ending up with a 404.

<baris>

You need monitor cpu/ram usage and network traffic. What version of nodebb are you running? 3.7.0 had a memory leak that was fixed in 3.7.1. Make sure you are on that version. If you have lots of traffic you will have to run multiple nodebb processes and load balance with a proxy.

Tomáš Nesrovnal

Hi Baris,
thanks for help. We finally figured it out. It was a simple error in our plugin that caused restarting NodeBB.

We are running the latest version - v3.7.1 and the forums runs in multiple nodebb processes.

Tomáš Nesrovnal

Hi again, @baris and @julian
the plugin causing restarts wasn't the only problem. I found another problem and I think it's performance related.

When a user is deleted (with API or directly in the forum), it seems that the request will block the event loop or database for a longer time than our health probe timeout allows. This is causing more restarts of our nodebb proceses.

We have a lot of users and data. The recent database upgrade we did from version 2.x to 3.x took several hours.

I turned on 'debug' log level and there are still no errors in the logs.

When we temporarily disabled user deletion, the forum doesn't restart anymore. So I hope this is the last problem after the upgrade.

Looking into the code, the process of deleting user is doing many things. Do you think that there might be something blocking the event loop when there is a really of of data of all sorts? Or do you have any tips what to check?

Thanks
Tomas

<baris>

I guess it depends on the user being deleted. A lot of the code in user deletion is async db calls though so there isn't alot of code there that blocks the nodejs event loop. You should take a look at your db utiliization while user delete is happening and see if that's the problem.

Tomáš Nesrovnal

Thanks Baris. I tracked down the particular command that is causing troubles: The glob that is looking for pictures and covers to delete from the /assets/uploads/profile directory takes too long.

A quick fix will be to delete only user's cover:url (if it's a file not url) and uploadedpicture from the directory. But it won't work in general when keepAllUserImages is true. The best fix would be to store all paths to user covers and pictures and then delete them all without needing to use glob.

What do you think?

<baris>

@Tomáš-Nesrovnal Yeah that makes sense with a ton of files in that folder that could slow things down, we would have to track the cover and profile upload filenames for each user.

o o

why not prefix files in a user dir? /assets/$user-id/

Tomáš Nesrovnal

I created a bug report here: https://github.com/NodeBB/NodeBB/issues/12449

<baris>

Yeah I think having a folder under public/uploads/profile for each user would make deleting easier. We can get rid of the glob and delete that folder on user deletion. But we would have to write an upgrade script to move all files and update the path in the user object since they are stored as "/assets/uploads/profile/1-profileavatar-1709738685611.jpeg"