hundreds of polling requests

[exodo]

@julian said:

After discussion with @baris, I was mistaken about NodeBB's role with cluster management.

We rely purely on nginx (or apache) to forward the incoming requests to the correct NodeBB instance. If NodeBB is set up to listen to two ports, it will start two instances, but will not do any routing.

So, @TheBronx @KingCat @zack, you'll have to set up your environment as follows:

Domain Name -> CloudFlare -> Nginx -> NodeBB x2 (assuming ports 4567 and 4568).

Nginx will need to proxy to the upstream IP (even if it is the same machine), so you'll need to use an upstream block like so:

upstream io_nodes {
    ip_hash;
    server 127.0.0.1:4567;
    server 127.0.0.1:4568;
}

The ip_hash; part is important, so incoming IP addresses are sent to the same server during handshaking.

Long polling should work fine then.

i can confirm the hundreds of polling request its not a ws issue
tried it with this info

   "socket.io": {
        "transports" : ["polling"]
   }

so it only uses polling
i put only one port so will run one instance and will be ok, no error on console

but if you put several posts

"port": ["4567", "4568", "4569", "4570"],

with proxy on nginx

server {
        listen 80;

        server_name yournodebb.com;

        location / {
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $http_host;
            proxy_set_header X-NginX-Proxy true;

            proxy_pass http://io_nodes;
            proxy_redirect off;

            # Socket.IO Support
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }
    }
    
    upstream io_nodes {
        ip_hash;
        server 127.0.0.1:4567;
        server 127.0.0.1:4568;
        server 127.0.0.1:4569;
        server 127.0.0.1:4570;
    }

then you will get many request x second , notice domain isn't ws.exo.do and its only exo.do cause it's not running with web sockets.

[julian]

@zack Thanks for giving it a try Odd why the ip_hash directive isn't doing what it's supposed to. What nginx version are you running?

Are you using CloudFlare as well? You'll need to translate those IPs properly, as when going through CF, the requesting IP is actually one from CF's network, not the end user.

[exodo]

@julian said:

@zack Thanks for giving it a try Odd why the ip_hash directive isn't doing what it's supposed to. What nginx version are you running?

Are you using CloudFlare as well? You'll need to translate those IPs properly, as when going through CF, the requesting IP is actually one from CF's network, not the end user.

nginx/1.1.19 + Cloudfare

[pichalite]

@zack NodeBB documentation states minimum nginx version as v1.3.13. May be upgrading nginx will solve the issue?

https://docs.nodebb.org/en/latest/configuring/proxies/nginx.html

[exodo]

@pichalite said:

@zack NodeBB documentation states minimum nginx version as v1.3.13. May be upgrading nginx will solve the issue?

https://docs.nodebb.org/en/latest/configuring/proxies/nginx.html

worths a try

[exodo]

updated to nginx 1.8. and problem persist
notice this is not a cloudfare problem, browsing directly with server ip will end with same error if you put more than 1 port on node config , even with 1 port i´m getting 1 error 400 each minute.
if you click on one of them you get this
{"code":1,"message":"Session ID unknown"}

[KingCat]

Have you tried using nginx to serve the static assets? That's what fixed it for me: https://community.nodebb.org/topic/4146/socket-io-bad-requests-on-nodebb-0-6-1/12

[markkus]

@zack said:

updated to nginx 1.8. and problem persist
notice this is not a cloudfare problem, browsing directly with server ip will end with same error if you put more than 1 port on node config , even with 1 port i´m getting 1 error 400 each minute.
if you click on one of them you get this
{"code":1,"message":"Session ID unknown"}

I have exactly the same situation.

Edit: I changed config.json a little bit.

"socket.io": {
    "transports" : ["websocket"]
}

This seems to solve the polling errors in console (quite obvious lol), but the "lost connection to forum" still keeps appearing on the navbar every minute or so.

[TheBronx]

maybe we should ask the socket.io guys, it is really weird.
CloudFlare is clearly not the problem, as we have tried without it.
Nginx with ip_hash is not making any difference.

What's the problem then? has anyone ever configured NodeBB with 2 or more instances restricted to "transport":["polling"]? is it working without throwing tons of requests?

[TheBronx]

this sounds familiar to me: https://github.com/Automattic/socket.io/issues/2091
this other one might be related too: https://github.com/Automattic/socket.io/issues/1831

I'm going to create a simple test scenario, without nodebb, just socket.io and see if I can reproduce the error and open an issue.

[julian]

@TheBronx Please do.

We use two processes here, and use nginx with ip_hash; to go between each process, though we don't use polling exclusively.

[TheBronx]

Ok, this is the sample ws server: https://github.com/exo-do/misc/blob/master/multiple instance ws server/ws/index.js
I've started two servers, ports 9090 and 9091, and placed nginx on top of it, WITHOUT ip_hash. This is what happens:

Well, it is normal, long polling is jumping from one instance to the other. Now if I enable ip_hash:

So it seems there is no bug in socket.io long polling. Here is the nginx config I'm using in this test:

upstream io_nodes {
	ip_hash;
	server 127.0.0.1:9090;
	server 127.0.0.1:9091;
}
server {
		listen   8080; ## listen for ipv4; this line is default and implied
		listen   [::]:8080 default ipv6only=on; ## listen for ipv6

		server_name ws.blablablabla.com;
		
		location / {
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection "upgrade";
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			proxy_set_header Host $host;
			proxy_http_version 1.1;
			proxy_pass http://io_nodes;
		}
}

I've used the default config, and added just what socket.io says you have to add: http://socket.io/docs/using-multiple-nodes/#

We will have to check again and see if our nginx config is exactly the same I'm using here in the test.

[julian]

@TheBronx Might be the link in your post, but yeah... we sometimes do get false positives from Akismet

[TheBronx]

@julian said:

@TheBronx Might be the link in your post, but yeah... we sometimes do get false positives from Akismet

haha no problem, did the trick of posting some shit and then editing with the real message. Akismet, I won.

I can't find what's missing in our nginx conf. I want to try again without cloudflare, it should work...

[TheBronx]

I've been trying nginx configuration for hours but still can't make it work fine.
Some requests keep returning 400 Bad Request, with a message saying: "Session ID unknown"
Enabling the websocket log I've found that when this happens, the following message is printed in the log:
io: 0 emit [ 'disconnect', { reason: 'transport error', rooms: [ 'pLywexv6lX6jRFvcAAAA', 'online_guests' ] } ]

There are also some 502 Bad Gateway errors, but it all seems to be caused by Nginx not being correctly configured for NodeBB and long polling.

I'm not familiar with Nginx, which makes things harder. I've tried with a few configurations found here and there, like:

proxy_buffering off;
proxy_connect_timeout      60s;
proxy_send_timeout         60s;
proxy_read_timeout         60s;
keepalive_timeout          60s;
proxy_buffers 8 32k;
proxy_buffer_size 64k;
large_client_header_buffers 8 32k;

Some of them seem to increase 502 errors, decreasing bad requests, but that's all I can say, pretty useless.

What seems clear to me is:

• CloudFlare is not the problem (at least not right now), cause I'm testing without it, directly targeting Nginx.
• Socket.io alone (without NodeBB) doesn't seem to be the problem either, unless you remove ip_hash from upstreams.
• There is something between nginx and nodebb that is making long polling request to "timeout"?
• When I remove ip_hash from nginx upstreams in nodebb, then I start to see hundreds of requests, without sleeps in between. That's a good improvement!

What I'm noticing now is that whenever a request takes more than 10 seconds to respond, it fails with a bad request. But when the request gets answered before 10s, it works:

We are getting closer, I hope

[baris]

@TheBronx Hey that 10s leads me to believe it might be this piece of code.

https://github.com/NodeBB/NodeBB/blob/master/src/webserver.js#L44-L46

Try commenting it out or increasing it and see if it works.

[TheBronx]

@baris said:

@TheBronx Hey that 10s leads me to believe it might be this piece of code.

https://github.com/NodeBB/NodeBB/blob/master/src/webserver.js#L44-L46

Try commenting it out or increasing it and see if it works.

WHAT THE FUCK IS THAAAAAT haha, it works!!!!!!!!!!!!!!!!!!!!!!!!!
you guys are gonna pay for this xD

[baris]

Does it work with a value like 30? I'll increase it in core.

[TheBronx]

@baris said:

Does it work with a value like 30? I'll increase it in core.

it seems 30000 would be a good limit. anyways, that "magic number" should not be there hehe.

PD: thanks for the help guys! I would have spent a lot of hours configuring nginx @baris haha.

[baris]

@TheBronx If we don't specify it at all the default is 60seconds I believe.