config.json cluster change

<baris>

Made some changes to the way nodebb works in order to fix socket.io on cluster.

This only effects people who are using the "cluster" setting in config.json.

cluster setting has been removed in latest master, in order to spawn more than one nodebb process you have to add a port property into config.json. I'll post some samples below.

NodeBB with a single process. Listening on port 4567, no port property required.

{
    "url": "http://yournodebb.com:4567",
    "secret": "secretsauce",
    "database": "redis",
    "redis": {
        "host": "127.0.0.1",
        "port": "6379",
        "username": "",
        "password": "",
        "database": "0"
    }
}

NodeBB with 4 processes on ports 4567, 4568, 4569, 4570

{
    "url": "http://yournodebb.com",
    "port": ["4567", "4568", "4569", "4570"],
    "secret": "secretsauce",
    "database": "redis",
    "redis": {
        "host": "127.0.0.1",
        "port": "6379",
        "username": "",
        "password": "",
        "database": "0"
    }
}

If you specify multiple ports you need to setup a load balancer like nginx to proxy requests to these ports. Here is a sample nginx.conf that does that for the above config.json

server {
        listen 80;

        server_name yournodebb.com;

        location / {
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $http_host;
            proxy_set_header X-NginX-Proxy true;

            proxy_pass http://io_nodes;
            proxy_redirect off;

            # Socket.IO Support
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }
    }
    
    upstream io_nodes {
        ip_hash;
        server 127.0.0.1:4567;
        server 127.0.0.1:4568;
        server 127.0.0.1:4569;
        server 127.0.0.1:4570;
    }

The above nginx config will proxy requests to nodebb instances based on their ip, this is important for socket.io since polling doesn't work if requests during handshake go to different processes.

relevant commits
https://github.com/NodeBB/NodeBB/commit/64e13df14cd8ca5364e5e465926d49b1c2ca2654
https://github.com/NodeBB/NodeBB/commit/d62cdd5127bde78ba4b5c1f5e50ab05dc8ccd519

Let me know if you have any questions.

wzrdtales

Well, this wont work if you're behind a proxy with NGINX itself so far.

ip_hash uses the client ip.

There is a patch so far and an announce that a future version introduce this feature (1.7.1).
http://wiki.nginx.org/HttpUpstreamRequestHashModule

With this the X-Forwarded-ip can be hashed instead. Just now anyone who needs this would need to use a compiled version with this patch or the mainline version.

One question aside, why did you break up the support/use of cluster?

<baris>

Using node-cluster behind nginx doesn't work as well. When cluster is used we need to grab the clients IP and route them to the correct worker process based on their IP address. Something like this https://github.com/elad/node-cluster-socket.io. Actually we implemented that first here. But there is no way to read the clients IP address in the master process, see this issue https://github.com/indutny/sticky-session/issues/6. And connection.remoteAddress doesn't give the correct IP behind nginx.

It's worth mentioning that you don't have to use ip_hash if you just use websockets and disable all polling in socket.io.

If you want to use node cluster and socket.io 0.9.x we have that in the this branch https://github.com/NodeBB/NodeBB/tree/socket.io-0.9.x. It is essentially master with the only difference of using node-cluster and socket.io 0.9.x.

wzrdtales

@baris Well yes this is, of course, a problem. This can't work because the sticky-sessions working on Layer 3, but the Information we need is on Layer 4. Really I don't know why this haven't been realized by anyone till now. Seems like WebSockets in combination with node are not that widely used as I thought...

So far, you can give in your documentation a hint for everyone that they need the HttpUpstreamRequestHashModule if the NGINX itself is behind a proxy.

But this should be only a temporary solution, the good news.
I work right now on a solution and you will get a pull request soon

I think I will finish the sticky-session branch today and then I look into NodeBB, and hope that I can finish this today, too.

Edit: sticky-sessions are working now, but only from node version 0.9.6 to 0.11.14 yet, going to document everything and then open pull request. Over the next our I'm going to look at NodeBB.
https://github.com/wzrdtales/sticky-session/tree/v0.2.0
https://github.com/indutny/sticky-session/pull/17

Right now looking into NodeBB.

Edit:
Ok was away some hours, first impressions. Wow, you really should add more comments to your code.

wzrdtales

Finally the pull request is open https://github.com/NodeBB/NodeBB/pull/2527.

I'm out for today guys, hope you get everything. If not just ask, gn8.