hundreds of polling requests

TheBronx

What??? So this is the expected behavior? We set socketio.transports to ["polling"] thinking that would work
That's bad, you can't have a production board working like this, reconnecting again and again, doing hundreds of requests per minute and user.

@ApfelUser Yes, Cloudflare is blocking some of those requests, but a lot of them still reach the server, and that is only a single tab of a single user in a low traffic moment. In a heavy traffic day that would be a lot of load. And for what? For an instable connection that is not working correctly...

You are right, the plan is to create a secondary server, bypassing cloudflare to allow websockets, and modify the socketio.address parameter in config.json to point to that secondary server.
That should work (we hope), until attacked. In case an attacker targets the server by IP, we have to move to another server, turn on cloudflare temporarily and bypass again when the attack finishes. So we still need long polling to keep the forum working under attack. As a fallback option.

I'm with @pitaj and @zack, long polling should work.
I'm a bit disappointed, I was pretty sure this was a problem or misconfiguration on our side, but not the expected behavior. You guys did a really good job with NodeBB, I've said that before, and the API first approach is perfect also. Websockets are the future, but DDoS attacks are going to be there too. Long polling support is critical.

julian

Guys, long polling does work. If your browser or ISP doesn't support the websocket protocol, NodeBB will drop down to xhr-polling, and all functionality should continue working.

Now, the exception is that websockets doesn't work well with CloudFlare, and somehow socket.io cannot realise this and continues to try to upgrade the connection to websocket.

So the approach here is as always:

Use cloudflare as DNS only, without acceleration.

Grey clouds, over orange.

exodo

@julian i have my domain on grey cloud and still receiving hundreds of console log errors about web sockets, don't think that is the solution

TheBronx

Thanks for the answer @julian, but "grey clouds" (or bypass) is the same as removing cloudflare in terms of security: no DDoS protection at all.
We use Cloudflare as a security measure, cause it hides our server behind CloudFlare's IPs, thus removing the option to target our server directly with a DDoS attack.
If we use Cloudflare as DNS only, then everyone will know our IP, and that would be bad hehe.

We will keep investigating the issue, hope we can find the reason long polling is not working properly. I'm not sure it is Cloudflare, nginx is also in the middle, and NodeBB in the end, a lot of things we have to check.

julian

@TheBronx Please double check with CF, but I believe that is incorrect.

Bypassing CloudFlare via grey clouds just bypasses the acceleration and caching provided by CF. Their DDoS protection works on the DNS level, and you will still be using CloudFlare for that, and they will still be cloaking your IP address.

TheBronx

just ping a bypassed domain, and you will see the real IP address, not a cloudflare's one.
an attacker can send traffic to the IP, without even reaching cloudflare servers.
for example, a DNS amplification attack:
Imagine our server is on 1.2.3.4
An attacker sends DNS queries to vulnerable DNS resolvers, spoofing your IP address:

Hello DNS resolver, I'm 1.2.3.4, please, tell me all you know about nasa.gov
Server 1.2.3.4 here is your information!

Send a few thousand requests like that, and 1.2.3.4 is fucked. CloudFlare? CloudFlare knows nothing about this attack, nobody even asked him anything.

julian

Ah, I stand corrected, then. Unfortunate. Have you tried incapsula? They have a free plan, but I honestly do not know if they proxy websockets on their free plan as well.

exodo

@julian incapsula only support websockets on enterprise plans (same as cloudfare), they confirmed it to me by contact support.
cloudfare reveals ip on grey cloud for sure

TheBronx

@zack is right about Incapsula. And long polling also works pretty bad.
We will have to try without CF and without nginx to debug each part individually.

Thanks for the help guys.

julian

The site works with xhr-polling, to my knowledge. We had a client using xhr-polling for awhile, with no loss of functionality...

There are many ajax requests, yes, but that is the nature of long polling, no? Then again there shouldn't be multiple calls per second... just one or two every 10s or so...

markkus

Perhaps there can be something done with SRV records (for spoofing IP)? As far as I'm concerned, when your DNS record has "grey clouds" in CloudFlare, it directs to your IP directly.

julian

Some more thoughts on the issue:

When you don't have websockets working, we drop down to xhr polling (as established prior).
To establish a secure connection, socket.io goes through a multi-stage handshake to exchange information
If you are utilising multiple NodeBB processes, separate parts of the handshake can end up going to different processes, and the handshake is lost.

@markkus @TheBronx Can you try reducing the number of ports used by NodeBB to just 1, and see if the issue persists? I have a feeling it will not, though of course, this is not a solution...

TheBronx

@julian you are right, when setting only one instance, long polling works as expected. It only uses one request at a time, that gets an answer soon or late, but it no longer uses 4 or 5 request at the same time. there are also no 400 errors.

we have to go back to multiple instances, of course, but it seems you have found something. is there anything we can do to make nginx redirect all requests from the same IP (or user) to the same nodebb instance? this should help, am I wrong?

by the way, we have tried with a secondary server (we still have to fix some issues), and we have found that the wiki is wrong about the socketio configuration:
it says you have to create a block in config.json like:
"socketio": { ... }
when it should say:
"socket.io": { ... }
notice the dot xD

<baris>

@TheBronx Yes you should use the ip_hash; directive and redirect the clients to the same nodebb instance based on IP.

julian

@baris This would help if you only had one NodeBB process per ip, but I think if you had multiple, then each process uses its own in-memory space for socket.io handshaking, right?

Seems this is the problem.

julian

The solution for this problem (in addition to using ip_hash, that is) as suggested online seems to be to use the sticky-sessions module, although NodeBB actually doesn't use the cluster module, so this is not an applicable solution unless we rewrite the load balancing code in loader.js to use Node.js' cluster module again.

KingCat

@TheBronx Have you tried setting up a page rule to disable acceleration, caching, and apps? That way you can still use the WAF and be hidden behind the CloudFlare IP but maybe web sockets will work better...

exodo

That wont work as cf dont dupport websocket on lower plans, no matjer if you disable catching, etc

@julian just tried websockets with a second server with another nodebb instance and the socket.io from the main server with cloudfare refering that second ip.
It works but it doesnt let to login or post.
How do you share "cookies" or logged in status between that 2 node instances?

julian

After discussion with @baris, I was mistaken about NodeBB's role with cluster management.

We rely purely on nginx (or apache) to forward the incoming requests to the correct NodeBB instance. If NodeBB is set up to listen to two ports, it will start two instances, but will not do any routing.

So, @TheBronx @KingCat @zack, you'll have to set up your environment as follows:

Domain Name -> CloudFlare -> Nginx -> NodeBB x2 (assuming ports 4567 and 4568).

Nginx will need to proxy to the upstream IP (even if it is the same machine), so you'll need to use an upstream block like so:

upstream io_nodes {
    ip_hash;
    server 127.0.0.1:4567;
    server 127.0.0.1:4568;
}

The ip_hash; part is important, so incoming IP addresses are sent to the same server during handshaking.

Long polling should work fine then.

exodo

@julian said:

After discussion with @baris, I was mistaken about NodeBB's role with cluster management.

We rely purely on nginx (or apache) to forward the incoming requests to the correct NodeBB instance. If NodeBB is set up to listen to two ports, it will start two instances, but will not do any routing.

So, @TheBronx @KingCat @zack, you'll have to set up your environment as follows:

Domain Name -> CloudFlare -> Nginx -> NodeBB x2 (assuming ports 4567 and 4568).

Nginx will need to proxy to the upstream IP (even if it is the same machine), so you'll need to use an upstream block like so:
upstream io_nodes {
    ip_hash;
    server 127.0.0.1:4567;
    server 127.0.0.1:4568;
}
The ip_hash; part is important, so incoming IP addresses are sent to the same server during handshaking.

Long polling should work fine then.

not sure how to do this
how you run 2 instances of node with same domain?