config.json cluster change

[baris]

Made some changes to the way nodebb works in order to fix socket.io on cluster.

This only effects people who are using the "cluster" setting in config.json.

cluster setting has been removed in latest master, in order to spawn more than one nodebb process you have to add a port property into config.json. I'll post some samples below.

NodeBB with a single process. Listening on port 4567, no port property required.

{
    "url": "http://yournodebb.com:4567",
    "secret": "secretsauce",
    "database": "redis",
    "redis": {
        "host": "127.0.0.1",
        "port": "6379",
        "username": "",
        "password": "",
        "database": "0"
    }
}

NodeBB with 4 processes on ports 4567, 4568, 4569, 4570

{
    "url": "http://yournodebb.com",
    "port": ["4567", "4568", "4569", "4570"],
    "secret": "secretsauce",
    "database": "redis",
    "redis": {
        "host": "127.0.0.1",
        "port": "6379",
        "username": "",
        "password": "",
        "database": "0"
    }
}

If you specify multiple ports you need to setup a load balancer like nginx to proxy requests to these ports. Here is a sample nginx.conf that does that for the above config.json

server {
        listen 80;

        server_name yournodebb.com;

        location / {
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $http_host;
            proxy_set_header X-NginX-Proxy true;

            proxy_pass http://io_nodes;
            proxy_redirect off;

            # Socket.IO Support
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }
    }
    
    upstream io_nodes {
        ip_hash;
        server 127.0.0.1:4567;
        server 127.0.0.1:4568;
        server 127.0.0.1:4569;
        server 127.0.0.1:4570;
    }

The above nginx config will proxy requests to nodebb instances based on their ip, this is important for socket.io since polling doesn't work if requests during handshake go to different processes.

relevant commits
https://github.com/NodeBB/NodeBB/commit/64e13df14cd8ca5364e5e465926d49b1c2ca2654
https://github.com/NodeBB/NodeBB/commit/d62cdd5127bde78ba4b5c1f5e50ab05dc8ccd519

Let me know if you have any questions.

0 barisusakli committed to NodeBB/NodeBB

removed node cluster spawn procs on ports

0 barisusakli committed to NodeBB/NodeBB

added support for specifying which ports to use

if no port is specified in config.json port from url will be used if no
port in url then default to 4567

port in config.json can be an array or a single port number

[wzrdtales]

Well, this wont work if you're behind a proxy with NGINX itself so far.

ip_hash uses the client ip.

There is a patch so far and an announce that a future version introduce this feature (1.7.1).

Welcome to NGINX Wiki! | NGINX

With this the X-Forwarded-ip can be hashed instead. Just now anyone who needs this would need to use a compiled version with this patch or the mainline version.

One question aside, why did you break up the support/use of cluster?

[baris]

Using node-cluster behind nginx doesn't work as well. When cluster is used we need to grab the clients IP and route them to the correct worker process based on their IP address. Something like this https://github.com/elad/node-cluster-socket.io. Actually we implemented that first here. But there is no way to read the clients IP address in the master process, see this issue https://github.com/indutny/sticky-session/issues/6. And connection.remoteAddress doesn't give the correct IP behind nginx.

It's worth mentioning that you don't have to use ip_hash if you just use websockets and disable all polling in socket.io.

If you want to use node cluster and socket.io 0.9.x we have that in the this branch https://github.com/NodeBB/NodeBB/tree/socket.io-0.9.x. It is essentially master with the only difference of using node-cluster and socket.io 0.9.x.

vanwagonet created this issue in indutny/sticky-session

open Is remoteAddress going to always be the same for applications behind a proxy? #6

[wzrdtales]

@baris Well yes this is, of course, a problem. This can't work because the sticky-sessions working on Layer 3, but the Information we need is on Layer 4. Really I don't know why this haven't been realized by anyone till now. Seems like WebSockets in combination with node are not that widely used as I thought...

So far, you can give in your documentation a hint for everyone that they need the HttpUpstreamRequestHashModule if the NGINX itself is behind a proxy.

But this should be only a temporary solution, the good news.
I work right now on a solution and you will get a pull request soon

I think I will finish the sticky-session branch today and then I look into NodeBB, and hope that I can finish this today, too.

Edit: sticky-sessions are working now, but only from node version 0.9.6 to 0.11.14 yet, going to document everything and then open pull request. Over the next our I'm going to look at NodeBB.
https://github.com/wzrdtales/sticky-session/tree/v0.2.0
https://github.com/indutny/sticky-session/pull/17

Right now looking into NodeBB.

Edit:
Ok was away some hours, first impressions. Wow, you really should add more comments to your code.

wzrdtales created this issue in indutny/sticky-session

open Support of Layer 4 Hash Balancing, fixes several issues and supporting reverse proxy ips #17

[wzrdtales]

Finally the pull request is open https://github.com/NodeBB/NodeBB/pull/2527.

I'm out for today guys, hope you get everything. If not just ask, gn8.

wzrdtales created this issue in NodeBB/NodeBB

closed Enabling cluster, sticky-sessions, level 4 based hash balancing #2527