• Hi there!

    I have very high load in our server because NodeBB traffic. The forum y very low used, 10 - 20 post daily, but spiders and websockets connections between forum and clients, causes high traffic, and trigger a lot of MongoDB queries.

    Are there a way to mitigate this, such as web proxies (varnish), etc, without disabling crawler indexing? I see in docs the varnish configuration, but seems that is not cache.

    Regards!
    Normando

  • Global Moderator Plugin & Theme Dev

    Are you using nginx to serve static assets?


  • @pitaj said in How to reduce the load?:

    Are you using nginx to serve static assets?

    Yes. This is my conf

    upstream io_nodes {
        ip_hash;
        server 127.0.0.1:20000;
        server 127.0.0.1:20001;
    }
    
    server {
        listen 192.168.10.10:80;
        server_name vapeandoargentina.com.ar;
        return 301 https://$server_name$request_uri;
    }
    
    server {
        listen 192.168.10.10:443 ssl http2;
        server_name vapeandoargentina.com.ar;
    
    #    access_log  /var/log/nginx/vapor-access.log;
        access_log off;
        error_log   /var/log/nginx/vapor-error.log;
        ssl_certificate /etc/letsencrypt/live/vapeandoargentina.com.ar/fullchain.pem; # managed by Certbot
        ssl_certificate_key /etc/letsencrypt/live/vapeandoargentina.com.ar/privkey.pem; # managed by Certbot
    
        add_header X-Frame-Options "SAMEORIGIN";
        add_header X-Download-Options noopen;
        add_header X-Permitted-Cross-Domain-Policies none;
    
        proxy_set_header X-Forwarded-Proto $scheme;
    
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;
        proxy_redirect off;
    
        # Socket.io Support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    
      proxy_buffers 4 256k;
      proxy_buffer_size 128k;
      proxy_busy_buffers_size 256k;
      proxy_read_timeout 180s;
      proxy_send_timeout 180s;
    
        gzip            on;
        gzip_min_length 1000;
        gzip_proxied    off;
        gzip_types      text/plain application/xml application/x-javascript text/css application/json;
    
        location @nodebb {
            proxy_pass http://io_nodes;
            #proxy_pass http://127.0.0.1:6081;
        }
    
        location ~ ^/assets/(.*) {
            root /opt/vapor/nodebb/;
            try_files /build/public/$1 /public/$1 @nodebb;
        }
    
        location /plugins/ {
            root /opt/vapor/nodebb/build/public/;
            try_files $uri @nodebb;
        }
    
        location / {
            proxy_pass http://io_nodes;
            #proxy_pass http://127.0.0.1:6081;
        }
    }
    
    
  • Community Rep

    @normando Hmm.. curious... how many worker processes and connections?

    nginx -T| grep -i worker_connect
    nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
    nginx: configuration file /etc/nginx/nginx.conf test is successful
        worker_connections  1024;
    
    [root@forums nginx]# nginx -T| grep -i worker_process
    nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
    nginx: configuration file /etc/nginx/nginx.conf test is successful
    worker_processes  auto;
    
    
    

    I am not all that familiar with nginx's proxy tunables. And some of above differs from my set up. Be that as it may, why not spin up a couple more nodebb nodes, i.e. 20002 and 20003? You've only got two. Just to see what happens whilst yer' testing anyways, eh?

    Beyond tweakin' and tunin' nginx et.al. you are effectively experiencing a low level denial of service. So you much take steps to mitigate at the problem source. Maybe some adaptive firewall rulesets? Maybe a WAF? Maybe Fail2Ban?

    Have fun! o/ 🐕


  • @gotwf I will be fun with DoS 😖

    Anyway, seems crawler bots.

    Yes, I get the same as you;

    # nginx -T| grep -i worker_connect
    nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
    nginx: configuration file /etc/nginx/nginx.conf test is successful
        worker_connections 1024;
    # nginx -T| grep -i worker_process
    nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
    nginx: configuration file /etc/nginx/nginx.conf test is successful
    worker_processes auto;
    

    I will test increasing nodes, but still thinking how to cache the output.
    Thanks


  • @normando Wondering if this will help. I use caching over at https://sudonix.com and it's set as follows

    server {
    	listen x.x.x.x;
    	listen [::]:80;
    	server_name www.sudonix.com sudonix.com;
    	return 301 https://sudonix.com$request_uri;
    	access_log /var/log/virtualmin/sudonix.com_access_log;
    	error_log /var/log/virtualmin/sudonix.com_error_log;
    }
    server {
    	listen x.x.x.x:443 ssl http2;
            server_name  www.sudonix.com;
    	ssl_certificate /home/sudonix/ssl.combined;
    	ssl_certificate_key /home/sudonix/ssl.key;
    	return 301 https://sudonix.com$request_uri;
    	access_log /var/log/virtualmin/sudonix.com_access_log;
    	error_log /var/log/virtualmin/sudonix.com_error_log;
    }
    server {
    	server_name sudonix.com;
    	listen x.x.x.x:443 ssl http2;
    	access_log /var/log/virtualmin/sudonix.com_access_log;
    	error_log /var/log/virtualmin/sudonix.com_error_log;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;
        proxy_redirect off;
        # Socket.io Support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        gzip            on;
        gzip_min_length 1000;
        gzip_proxied    off;
        gzip_types      text/plain application/xml text/javascript application/javascript application/x-javascript text/css application/json;
        location @nodebb {
            proxy_pass http://127.0.0.1:4567;
        }
        location ~ ^/assets/(.*) {
            root /home/sudonix/nodebb/;
            try_files /build/public/$1 /public/$1 @nodebb;
            add_header Cache-Control "max-age=31536000";
        }
        location /plugins/ {
            root /home/sudonix/nodebb/build/public/;
            try_files $uri @nodebb;
            add_header Cache-Control "max-age=31536000";
        }
        location / {
            proxy_pass http://127.0.0.1:4567;
        }
        add_header X-XSS-Protection "1; mode=block";
        add_header X-Download-Options "noopen" always;
        add_header Content-Security-Policy "upgrade-insecure-requests" always;
        add_header Referrer-Policy 'no-referrer' always;
        add_header Permissions-Policy "accelerometer=(), camera=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), payment=(), usb=()" always;
        add_header X-Powered-By "Sudonix" always;
        add_header Access-Control-Allow-Origin "https://sudonix.com https://blog.sudonix.com" always;
        add_header X-Permitted-Cross-Domain-Policies "none" always;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        ssl_certificate /home/sudonix/ssl.combined;
        ssl_certificate_key /home/sudonix/ssl.key;
        rewrite https://sudonix.com/$1 break;
    	if ($scheme = http) {
    		rewrite ^/(?!.well-known)(.*) https://sudonix.com/$1 break;
    	}
    }
    
    
  • Community Rep

    @normando said in How to reduce the load?:

    Anyway, seems crawler bots.

    Do you have the logs? Is it primarily googlebot? It will come in like a thundering herd of thunderin' hoovers. Does not respect conventions, e.g. robots.txt, but rather requires you to sign up for their webmaster tools. Anyways, back to the relevance, if not, then googlebot will basically scale to suck up as much as it can as fast as it possibly can. Hence, if you are running NodeBB on a VM in some reasonably well connected data center, network bandwidth is unlikely to be the bottleneck, leaving the vm itself.

    The googlebot comes in fast, but also leaves fast. Several times a day, at +/- regular intervals depending on site activity and its mystery algos. So if you are seeing more off the wall bots, at random times 'twixt visits, then maybe these bots warrant further investigation. Otherwise, not much you can do about goog bot. But maybe give it more resources and it can leave sooner.

    Have fun! 🐕


  • @phenomlab thanks! I will test adding cache headers.

  • Community Rep

    Another quick thought: What is ACP->Advanced->Cache telling you about your hit ratios, eh?

    Have fun! 🐕

  • GNU/Linux Admin

    I wonder if Googlebot would understand if you sent back HTTP 420 Enhance Your Calm 😆 used as a rate-limiting mechanism.


  • @julian said in How to reduce the load?:

    HTTP 420 Enhance Your Calm

    Lenina Huxley....


  • @julian said in How to reduce the load?:

    I wonder if Googlebot would understand if you sent back HTTP 420 Enhance Your Calm 😆 used as a rate-limiting mechanism.

    hahaahahah 😁


  • @gotwf said in How to reduce the load?:

    Another quick thought: What is ACP->Advanced->Cache telling you about your hit ratios, eh?

    Have fun! 🐕

    Now increasing 🙂

    529b6cf9-0739-4e9b-ac47-c1e513c6821a-image.png

    I will wait few days to check again. Thanks!

  • GNU/Linux Admin

    @phenomlab said in How to reduce the load?:

    @julian said in How to reduce the load?:

    HTTP 420 Enhance Your Calm

    Lenina Huxley....

    And that made me think of Brave New World, which I realize wasn't your intention, but now I want to read that book again...


  • @julian Yeah, mine was from from "Demolition Man"

Suggested Topics

  • 5
  • 6
  • 8
  • 11
  • 17
| |