@juan-g hey hey, thanks for the reply. I appreciate it. Guess I gotta take a wild guess at which tool to use and try my best. I knew about the load testing tools, was just hoping to get a hint as to what tool or configuration is used by the nodebb team.
Nonetheless: Thank you for caring 
I really do appreciate it.
I think this question does not belong in this forum. Try a general google first.
@Dravere thx for the reply. Sadly it couldn't be more true.
The reason why I chose this setup (for now, certainly won't stick around as I hate it) is because I have a Plesk installed to manage the standard websites on the server. Plesk comes with Apache and (at least in my plesk version) can put an nginx in front as a reverse proxy to all its features. Sadly the version that it brings with itself is an nginx < 1.3.13 as would be required for nodebb to be usable as a reverse proxy.
I have tried manually updating nginx but, as Plesk brings its own custom repo to pull its own images from, this is non-trivial and has rendered my setup unusable until it was reset. So currently I am stuck with Apache until I can get that figured out 
The only solution I could think of right now is to put a small machine in place that works as a sole loadbalancer and reverse proxy to which all domains point in the first place. That one can then distribute requests to the real server. That would enable me to run an additional nginx (not the plesk one) on a "non-80" port, allowing me to use any nginx. But as this is a sideproject that doesnt generate any income yet, putting a machine in front is not a viable option
The Server is running on full SSDs. I would need to ask them which ones specifically are being used if it's of relevance. I figured "as long as it's SSD, it should be fine" up until now. Let me know if I should inquire and I will.
Thanks again for your time. Cheers!
So...this took a long time to work on. I finally have some odd numbers and findings that i'd like to share.
Maybe you have some insight or interpretations for me:
I found the server's process landscape basically always like this:
What jumped my eye is:
Notices
The file limit is:
sudo lsof | wc -l
56353
While the maximum seems to be:
cat /proc/sys/fs/file-max
524288
So ulimit shouldn't be the problem. Right?
I was also able to see, using a load testing tool (Locust) that I finally came around setting up, that simulating 200 users (behaviour see #1 below) did not produce any exceptions UNTIL redis' bgsave started. While bgsave was running I was getting a 60-90% request-error percentage with the following failures:
# fails Method Name Type
8850 GET /topic/11705/testthema2 u'ConnectionError(MaxRetryError("HTTPSConnectionPool(host=\'schreibnacht.de\', port=443): Max retries exceeded with url: /topic/11705/testthema2 (Caused by NewConnectionError(\'<urllib3.connection.VerifiedHTTPSConnection object at 0x....>: Failed to establish a new connection: [Errno 61] Connection refused\',))",),)'
2870 GET /user/someUser/settings u'ConnectionError(MaxRetryError("HTTPSConnectionPool(host=\'schreibnacht.de\', port=443): Max retries exceeded with url: /user/someUser/settings (Caused by NewConnectionError(\'<urllib3.connection.VerifiedHTTPSConnection object at 0x....>: Failed to establish a new connection: [Errno 61] Connection refused\',))",),)'
1 GET /topic/11705/testthema2 u'ConnectionError(MaxRetryError("HTTPSConnectionPool(host=\'schreibnacht.de\', port=443): Max retries exceeded with url: /topic/11705/testthema2 (Caused by NewConnectionError(\'<urllib3.connection.VerifiedHTTPSConnection object at 0x....>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known\',))",),)'
2899 GET /user/janerow u'ConnectionError(MaxRetryError("HTTPSConnectionPool(host=\'schreibnacht.de\', port=443): Max retries exceeded with url: /user/janerow (Caused by NewConnectionError(\'<urllib3.connection.VerifiedHTTPSConnection object at 0x....>: Failed to establish a new connection: [Errno 61] Connection refused\',))",),)'
2 GET /topic/11705/testthema2 u'ConnectionError(MaxRetryError("HTTPSConnectionPool(host=\'schreibnacht.de\', port=443): Max retries exceeded with url: /topic/11705/testthema2 (Caused by NewConnectionError(\'<urllib3.connection.VerifiedHTTPSConnection object at 0x....>: Failed to establish a new connection: [Errno 60] Operation timed out\',))",),)'
1 GET /user/janerow u"ConnectionError(ProtocolError('Connection aborted.', error(54, 'Connection reset by peer')),)"
2949 GET / u'ConnectionError(MaxRetryError("HTTPSConnectionPool(host=\'schreibnacht.de\', port=443): Max retries exceeded with url: / (Caused by NewConnectionError(\'<urllib3.connection.VerifiedHTTPSConnection object at 0x....>: Failed to establish a new connection: [Errno 61] Connection refused\',))",),)'
1 GET /topic/11705/testthema2 u"ConnectionError(ProtocolError('Connection aborted.', error(54, 'Connection reset by peer')),)"
2 GET /topic/11705/testthema2 u'ConnectionError(ProtocolError(\'Connection aborted.\', BadStatusLine("\'\'",)),)'
1 GET /user/janerow u'ConnectionError(ProtocolError(\'Connection aborted.\', BadStatusLine("\'\'",)),)'
Deductions I made myself (pls correct if wrong)
Further actions
The next obvious thing is to get the Apache cluster working with correct load balancing between the 4 processes.
Next I need to understand why redis bgsave is killing the forum. It might just be that both run on the same core and redis uses this core 100%, so nothing else can use it in the meantime? Any other suggestions very welcome.
Additions
#1 - The simulated users are logging in, then heavily refresh the index page, a good number of forum posts, their profile and their settings page, then log out
#2 - Apache configuration I was playing with:
<Proxy balancer://nodebbcluster>
# node process 4567
BalancerMember http://x.x.x.x:4567
# node process 4568
BalancerMember http://x.x.x.x:4568
# node process 4569
BalancerMember http://x.x.x.x:4569
</Proxy>
And then :
ProxyPass / balancer://nodebbcluster/
With nodebb configuration:
{
"url": "https://www.schreibnacht.de",
"port": ["4567", "4568", "4569"],
"secret": "...",
"database": "redis",
"redis": {
"host": "127.0.0.1",
"port": "6379",
"password": "...",
"database": "0"
},
"secure": true
}
@yariplus What would a good ulimit be for nodebb? (or even for the described system above?)
Oh my...! Thank you for all the replies. And especially julian taking the time! Amazing, I check all of that right now and ss -s next time we have an event.
Will report back asap!
A word of sorry ahead. This is going to get lengthy. If you will, hear me out though as I am clueless as to how to progress.
I run https://schreibnacht.de. We host a monthly event where a renowned author joins the community for a Q&A for an hour. During this hour the forum slows down to being unusable with loading a page taking long than the timeout, some requests being completely cancelled and returning a 503 and some request going lightning fast, returning in less than a second.
Which requests return fast is random and does not have a pattern. 95% of all requests load within tens of seconds or run timeout/503.
The nodebb instance is a single one running on 6 vCores with 16 GB RAM with Redis as DB on Ubuntu 16.04. The system load is at 0.4/0.32/0.27 very constantly. RAM is 8.9 GB / 16 used, CPU usage is minimal outside of Redis' bg-save actions where one Core is used and the others keep idling. 336 MB of 1GB in swap. We use Apache as a proxy between the clients and nodebb.
Stopping the forum via ./nodebb stop, stopping redis (shutdown save) and then restarting redis then the forum does not change the situation which is one of the weirdest parts about it.
During our events we have around 50-150 concurrent users, depending on how renowned the special guest is. (So let's face it, it SHOULDN'T have any problems at all).
I desperately need some help identifying bottlenecks, hints for analysis where I can start looking.
I will go so far as to ping @julian directly, maybe you have some insight althought I know you are very busy.
I could bet my pants and shoes that it's some misconfiguration or some error happing somewhere between browser, Apache, Nodebb and Redis but I just don't know my way around them enough to find the hairpin in the haystack.
Thank you for all your help and time guys.
@juan-g hey hey, thanks for the reply. I appreciate it. Guess I gotta take a wild guess at which tool to use and try my best. I knew about the load testing tools, was just hoping to get a hint as to what tool or configuration is used by the nodebb team.
Nonetheless: Thank you for caring I really do appreciate it.
I am still very much in need of this. Any help possible?
bump
I think this question does not belong in this forum. Try a general google first.
In the meantime I found:
Testing
Run NodeBB in development mode:./nodebb dev
This will expose the plugin debug logs, allowing you to see if your plugin is loaded, and its hooks registered. Activate your plugin from the administration panel, and test it out.
From https://docs.nodebb.org/development/plugins/
This might also be helpful.
@pitaj thank you so much for your reply. The response noting
ProxyPass / http://backendserver:8080/ retry=0
Might be worth investigating. I was not aware there was a delay before determining a proxy-action as failed. Although it makes obvious sense something like that would exist.
Do you have any recommendation how I might load-test the nodebb instance? Any tool that you have used or have seen used effectively?
I know about various tools with which one can test a web application, but I might not be aware of any obvious pitfalls that some tools might have with nodeBB or Proxy architectures in general.
If I could create a testing scenario to reproduce the error, I might be able to verify if that fix helps. Otherwise I'll have to wait until our next event and potentially realize it hasn't helped at all
This is my Apache config btw, if it's informative in any way:
Protocols h2 http/1.1
ServerName www.schreibnacht.de
ServerAlias schreibnacht.de
SSLEngine on
# Basic security headers
Header always set X-Content-Type-Options "nosniff"
Header always set X-Xss-Protection "1; mode=block"
# NodeBB header
RequestHeader set X-Forwarded-Proto "https"
# Static file cache
<FilesMatch "\.(ico|jpg|jpeg|png|gif|js|css)$">
<IfModule mod_expires.c>
ExpiresActive on
ExpiresDefault "access plus 14 days"
Header set Cache-Control "public"
</IfModule>
</FilesMatch>
ProxyRequests off
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
# Custom Error Document when NodeBB is offline
ProxyPass /error-documents !
ErrorDocument 503 /error-documents/503.html
Alias /error-documents /path/to/public
# Websocket passthrough
RewriteEngine On
RewriteCond %{REQUEST_URI} ^/socket.io [NC]
RewriteCond %{QUERY_STRING} transport=websocket [NC]
RewriteRule /(.*) ws://localhost:4567/$1 [P,L]
ProxyPass / http://localhost:4567/ retry=0
ProxyPassReverse / http://localhost:4567/ retry=0
# Log stuff
ErrorLog ${APACHE_LOG_DIR}/www-schreibnacht-error.log
CustomLog ${APACHE_LOG_DIR}/www-schreibnacht-access.log combined
