nodebb slows down

Technical Support
  • This post is deleted!
  • @pichalite said in nodebb slows down:

    @sanatisharif try utilizing clustering to run NodeBB on multiple processes. Here is the link to docs on how to do it https://docs.nodebb.org/en/latest/scaling/index.html#utilise-clustering

    We did this . It has been so much helpful and we can see the different now. thanks lot

  • what about varnish cache ? how to clustering nodebb when we use varnish ?

    with nodeBB tutorials we have something like this for varnish:

    vcl 4.0;
    backend nodebb {
      .host = "127.0.0.1"; 
      .port = "4567"; 
        .connect_timeout = 1s;
        .first_byte_timeout = 2s;
        .between_bytes_timeout = 60s;
        .max_connections = 10000;
    }
    backend forum {                           
      .host = "127.0.0.1"; 
      .port = "8082"; 
      .connect_timeout = 5s;                  
      .first_byte_timeout = 30s;              
      .between_bytes_timeout = 60s;           
      .max_connections = 100000;              
    } 
    import std;
    sub vcl_recv {
    
            unset req.http.X-Forwarded-For;
            set req.http.X-Forwarded-For = client.ip;
            set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");
    
          # Pipe websocket connections directly to Node.js
          if (req.http.Upgrade ~ "(?i)websocket") {
             set req.backend_hint = nodebb;
             return (pipe);
          }
            if (req.http.host ~ "(?i)^(www.)?forum\.sanatisharif\.ir$")
            {
                    set req.backend_hint = forum;
                        if (req.url ~ "^/socket.io/") {
                          set req.backend_hint = nodebb;
                          return (pipe); 
                        }
                    return (pass);
            }
           ...
           ...
    }
    sub vcl_pipe {
      # Need to copy the upgrade header
      if (req.http.upgrade) {
        set bereq.http.upgrade = req.http.upgrade;
      }
    }
    ....
    
    

    nginx:

    upstream io_nodes {
        ip_hash;
        server 127.0.0.1:4567;
        server 127.0.0.1:4568;
        server 127.0.0.1:4569;
    }
    server {
        listen 8082;
    
        server_name community.nodebb.org;
    
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;
        proxy_redirect off;
    
        # Socket.io Support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    
        gzip            on;
        gzip_min_length 1000;
        gzip_proxied    off;
        gzip_types      text/plain application/xml application/x-javascript text/css application/json;
    
        location @nodebb {
            proxy_pass http://io_nodes;
        }
    
        location ~ ^/(images|language|sounds|templates|uploads|vendor|src\/modules|nodebb\.min\.js|stylesheet\.css|admin\.css) {
            root /path/to/nodebb/public/;
            try_files $uri $uri/ @nodebb;
        }
    
        location / {
            proxy_pass http://io_nodes;
        }
    }
    

    NodeBB config:

    {
        "port": ["4567", "4568", "4569"]  // will start three processes
    }
    
  • Actually we have two problems:

    a.The problem I mentioned at first which is about NodeBB performance . While NodeBB is running, it gets slower and slower till we have to restart it to make it fresh . In fact we have created a cron-job on our server for that . It restarts NodeBB every 6 hours and according to our experience when the time is close to a NodeBB restart, it has become really slow and it needs a restart vary badly.

    b.The second problem is about some NodeBB bugs that has been happening since we changed our Nginx configuration and ran it on three ports instead of just one port. After doing this we are facing a bunch of came-out-of-nowhere problems. Like sometimes when we open our forum's first page we can't see any categories (it is blank), or when we click on unseen posts it shows a route error instead. In the other words our forum doesn't work normally now and I'm sure you can imagine how frustrating it is. The tricky point here is that we also have Varnish. So there is NodeBB, Nginx and a Varnish in between which has been configured in the same way the NodeBB tutorials says. It basically passes every socket.io requests to NodeBB directly and every requests for a static file to the Nginx. But we have had Varnish for a while and we hadn't experienced any problems before we changes Nginx.

    Lastly, we appreciate any helps from you guys.
    Thanks in advance.

  • I just changed Varnish configuration.I added these three blocks:
    at the begining:

    sub vcl_init {                                                                                                                                                    
        new nodebb = directors.round_robin();                                                                                                                         
        nodebb.add_backend(nodebb01);                                                                                                                                 
        nodebb.add_backend(nodebb02);                                                                                                                                 
        nodebb.add_backend(nodebb03);                                                                                                                                 
    }  
    

    and in the middle:

    # Pipe websocket connections directly to Node.js                                                                                                            
          if (req.http.Upgrade ~ "(?i)websocket") {                                                                                                                   
             set req.backend_hint = nodebb.backend();                                                                                                                 
             return (pipe);                                                                                                                                           
          } 
    
    set req.backend_hint = forum;                                                                                                                     
    			if (req.url ~ "^/socket.io/") {                                                                                                               
    			  set req.backend_hint = nodebb.backend();                                                                                                    
    			  return (pipe); # return pass seems not working for websockets                                                                               
    			} 
    
  • What's the swap config like on your machine?

  • @julian I don't know, please tell me how can I know that?
    We are using debian 8 on SSD hard drive with 16GB memory and 24 core(intel xenon) cpu that serve multiple php websites and only one nodejs app (=NodeBB) .

    I have checked the server usage we are using 8G memory and 50% cpu usage at most!(in worse case )

  • @sanatisharif That is quite peculiar. I have not seen these symptoms before. NodeBB has always been fairly snappy and does not slow down with time...

  • @julian maybe nodebb acts like this because of varnish configs.

    I have changed it to hash directors instead of round_robin with 'client.identifier' as ID some of our problems gone but users complicate that some times our comunity is very slow!

    sub vcl_init {                                                                                                                                                    
        new nodebb = directors.hash();                                                                                                                         
        nodebb.add_backend(nodebb01);                                                                                                                                 
        nodebb.add_backend(nodebb02);                                                                                                                                 
        nodebb.add_backend(nodebb03);                                                                                                                                 
    }
    
    set req.backend_hint = nodebb.backend(client.identity);
    
  • Also when I checked varnish log dozens of request will be passed(MISS) so we can have better config for varnish.

  • @sanatisharif said in nodebb slows down:

    We are using debian 8 on SSD hard drive with 16GB memory and 24 core(intel xenon) cpu that serve multiple php websites and only one nodejs app (=NodeBB) .

    How many NodeBB processes are you running? Are they utilising a lot of CPU?

  • @julian said in nodebb slows down:

    @sanatisharif said in nodebb slows down:

    We are using debian 8 on SSD hard drive with 16GB memory and 24 core(intel xenon) cpu that serve multiple php websites and only one nodejs app (=NodeBB) .

    How many NodeBB processes are you running? Are they utilising a lot of CPU?

    I don't know, please tell me how can I know that?

  • You can check top for that.

  • @sanatisharif said in nodebb slows down:

    I don't know, please tell me how can I know that?

    $> top
    

    this will give you a bunch of statistics about the system, including that's running and how much CPU it is taking.

    to find out how many processes of nodejs are running i would recommend using

    #> ps aux |grep node
    

    this will spit our information about all processes that have "node" in their command line. you'll probably get a few false positives but you should be able to identify how many of those are nodebb easily enough.


Suggested Topics


  • NodeBB Upgrade Error

    Solved Technical Support
    0 Votes
    5 Posts
    190 Views

    No, I did not. It worked after I did so. Somehow I missed that obvious section...

  • 0 Votes
    5 Posts
    993 Views

    Please first check the data storage directories of your MongoDB. While doing some crazy tests and more or less intentionally made the MongoDB go crash I found at times that it "forgot" about the available data, even though there were still kind of big files probably with all the data. I don't know how to recover from there but I assume there would be some articles out there in the internet helping you with it.

    But yeah, mongodump, borg backup, lftp synchronize to remote drive (or similar). Every day. And if you're at it, make sure the backups on your local computer are working as intended as well 😉

  • apache/nodebb issues

    Technical Support
    0 Votes
    6 Posts
    2k Views

    thanks for the solution, issue is fixed now

  • 0 Votes
    13 Posts
    6k Views

    Final update, it worked on the production server!! so now our forum is working properly sending emails the Linux way on a Windows Server OS, quite nice I would say, just as a side note our forum is actually really small in number of users, as it is an internal forum, so I can't assure this setup will work out to other forum's expectations or needs, also I would like to add that this shouldn't be the way to do it on Windows as it is kind of a weird way to handle it so in later versions it would be nice to see it reworked or polished for Windows environments (especially because it looks like the program used to simulate sendmail on Windows is no longer receiving maintenance).

  • 0 Votes
    11 Posts
    4k Views

    @Juxtapo What do the MongoDB logs suggest is the problem?