nodebb slows down

[julian]

This post is deleted!

[sanatisharif]

@pichalite said in nodebb slows down:

@sanatisharif try utilizing clustering to run NodeBB on multiple processes. Here is the link to docs on how to do it https://docs.nodebb.org/en/latest/scaling/index.html#utilise-clustering

We did this . It has been so much helpful and we can see the different now. thanks lot

[sanatisharif]

what about varnish cache ? how to clustering nodebb when we use varnish ?

with nodeBB tutorials we have something like this for varnish:

vcl 4.0;
backend nodebb {
  .host = "127.0.0.1"; 
  .port = "4567"; 
    .connect_timeout = 1s;
    .first_byte_timeout = 2s;
    .between_bytes_timeout = 60s;
    .max_connections = 10000;
}
backend forum {                           
  .host = "127.0.0.1"; 
  .port = "8082"; 
  .connect_timeout = 5s;                  
  .first_byte_timeout = 30s;              
  .between_bytes_timeout = 60s;           
  .max_connections = 100000;              
} 
import std;
sub vcl_recv {

        unset req.http.X-Forwarded-For;
        set req.http.X-Forwarded-For = client.ip;
        set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");

      # Pipe websocket connections directly to Node.js
      if (req.http.Upgrade ~ "(?i)websocket") {
         set req.backend_hint = nodebb;
         return (pipe);
      }
        if (req.http.host ~ "(?i)^(www.)?forum\.sanatisharif\.ir$")
        {
                set req.backend_hint = forum;
                    if (req.url ~ "^/socket.io/") {
                      set req.backend_hint = nodebb;
                      return (pipe); 
                    }
                return (pass);
        }
       ...
       ...
}
sub vcl_pipe {
  # Need to copy the upgrade header
  if (req.http.upgrade) {
    set bereq.http.upgrade = req.http.upgrade;
  }
}
....

nginx:

upstream io_nodes {
    ip_hash;
    server 127.0.0.1:4567;
    server 127.0.0.1:4568;
    server 127.0.0.1:4569;
}
server {
    listen 8082;

    server_name community.nodebb.org;

    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-NginX-Proxy true;
    proxy_redirect off;

    # Socket.io Support
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";

    gzip            on;
    gzip_min_length 1000;
    gzip_proxied    off;
    gzip_types      text/plain application/xml application/x-javascript text/css application/json;

    location @nodebb {
        proxy_pass http://io_nodes;
    }

    location ~ ^/(images|language|sounds|templates|uploads|vendor|src\/modules|nodebb\.min\.js|stylesheet\.css|admin\.css) {
        root /path/to/nodebb/public/;
        try_files $uri $uri/ @nodebb;
    }

    location / {
        proxy_pass http://io_nodes;
    }
}

NodeBB config:

{
    "port": ["4567", "4568", "4569"]  // will start three processes
}

[sanatisharif]

Actually we have two problems:

a.The problem I mentioned at first which is about NodeBB performance . While NodeBB is running, it gets slower and slower till we have to restart it to make it fresh . In fact we have created a cron-job on our server for that . It restarts NodeBB every 6 hours and according to our experience when the time is close to a NodeBB restart, it has become really slow and it needs a restart vary badly.

b.The second problem is about some NodeBB bugs that has been happening since we changed our Nginx configuration and ran it on three ports instead of just one port. After doing this we are facing a bunch of came-out-of-nowhere problems. Like sometimes when we open our forum's first page we can't see any categories (it is blank), or when we click on unseen posts it shows a route error instead. In the other words our forum doesn't work normally now and I'm sure you can imagine how frustrating it is. The tricky point here is that we also have Varnish. So there is NodeBB, Nginx and a Varnish in between which has been configured in the same way the NodeBB tutorials says. It basically passes every socket.io requests to NodeBB directly and every requests for a static file to the Nginx. But we have had Varnish for a while and we hadn't experienced any problems before we changes Nginx.

Lastly, we appreciate any helps from you guys.
Thanks in advance.

[sanatisharif]

I just changed Varnish configuration.I added these three blocks:
at the begining:

sub vcl_init {                                                                                                                                                    
    new nodebb = directors.round_robin();                                                                                                                         
    nodebb.add_backend(nodebb01);                                                                                                                                 
    nodebb.add_backend(nodebb02);                                                                                                                                 
    nodebb.add_backend(nodebb03);                                                                                                                                 
}  

and in the middle:

# Pipe websocket connections directly to Node.js                                                                                                            
      if (req.http.Upgrade ~ "(?i)websocket") {                                                                                                                   
         set req.backend_hint = nodebb.backend();                                                                                                                 
         return (pipe);                                                                                                                                           
      } 

set req.backend_hint = forum;                                                                                                                     
			if (req.url ~ "^/socket.io/") {                                                                                                               
			  set req.backend_hint = nodebb.backend();                                                                                                    
			  return (pipe); # return pass seems not working for websockets                                                                               
			} 

[julian]

What's the swap config like on your machine?

[sanatisharif]

@julian I don't know, please tell me how can I know that?
We are using debian 8 on SSD hard drive with 16GB memory and 24 core(intel xenon) cpu that serve multiple php websites and only one nodejs app (=NodeBB) .

I have checked the server usage we are using 8G memory and 50% cpu usage at most!(in worse case )

[julian]

@sanatisharif That is quite peculiar. I have not seen these symptoms before. NodeBB has always been fairly snappy and does not slow down with time...

[sanatisharif]

@julian maybe nodebb acts like this because of varnish configs.

I have changed it to hash directors instead of round_robin with 'client.identifier' as ID some of our problems gone but users complicate that some times our comunity is very slow!

sub vcl_init {                                                                                                                                                    
    new nodebb = directors.hash();                                                                                                                         
    nodebb.add_backend(nodebb01);                                                                                                                                 
    nodebb.add_backend(nodebb02);                                                                                                                                 
    nodebb.add_backend(nodebb03);                                                                                                                                 
}

set req.backend_hint = nodebb.backend(client.identity);

[sanatisharif]

Also when I checked varnish log dozens of request will be passed(MISS) so we can have better config for varnish.

[julian]

@sanatisharif said in nodebb slows down:

We are using debian 8 on SSD hard drive with 16GB memory and 24 core(intel xenon) cpu that serve multiple php websites and only one nodejs app (=NodeBB) .

How many NodeBB processes are you running? Are they utilising a lot of CPU?

[sanatisharif]

@julian said in nodebb slows down:

@sanatisharif said in nodebb slows down:

We are using debian 8 on SSD hard drive with 16GB memory and 24 core(intel xenon) cpu that serve multiple php websites and only one nodejs app (=NodeBB) .

How many NodeBB processes are you running? Are they utilising a lot of CPU?

I don't know, please tell me how can I know that?

[julian]

You can check top for that.

[accalia]

@sanatisharif said in nodebb slows down:

I don't know, please tell me how can I know that?

$> top

this will give you a bunch of statistics about the system, including that's running and how much CPU it is taking.

to find out how many processes of nodejs are running i would recommend using

#> ps aux |grep node

this will spit our information about all processes that have "node" in their command line. you'll probably get a few false positives but you should be able to identify how many of those are nodebb easily enough.