hundreds of polling requests
-
Hi there guys!
We finally launched our NodeBB board a few days ago (exo.do), but we are having trouble with long polling.
We are behind Cloudflare (because of DDoS attacks) so we cannot use websockets yet. But long polling doesn't seem to be working properly, it is causing an enormous load on the server, due to the big amout of requests made per client:
just open the website, stay in the home page for a few seconds with the network console opened, and wait for the party to begin.
is that normal? what's going on there?Thank you!
EDIT: here is a gif also:
-
That is NodeBB trying to establish a websocket connection. It fails because you are using cloudflare and websockets are only available for enterprise customers. Normal long polling with HTTP/S GET and POST would actually work, but NodeBB is built for using websockets. You would need to modify the NodeBB core to stop using websockets, and this'd be a lot of work.
Also, I don't understand why your server is under heavy load? Cloudflare is just blocking these requests by returning a 400. Your root server should not be getting any of these requests.An option would be that you configure NodeBB to use a subdomain, e.g websockets.exo.do, and disable cloudflare from proxying this subdomain by clicking on the cloud-symbol so that it is grey.
-
I've said this before, I'll say it again.
Long polling needs to work. Make it work. You just can not depend on websockets for all of your functionality.
Please, enable long polling. Fix it.
-
@pitaj said:
I've said this before, I'll say it again.
Long polling needs to work. Make it work. You just can not depend on websockets for all of your functionality.
Please, enable long polling. Fix it.
totally agreed, setting array with long polling only on node config won't solve this
big forums can't at the moment run with ws due to ddos attacks
cloudfare or incapsula support websockets only on enterprise plans, thats hundreds of dollars/month
can @julian @psychobunny comment on this? -
What??? So this is the expected behavior? We set
socketio.transportsto["polling"]thinking that would work
That's bad, you can't have a production board working like this, reconnecting again and again, doing hundreds of requests per minute and user.@ApfelUser Yes, Cloudflare is blocking some of those requests, but a lot of them still reach the server, and that is only a single tab of a single user in a low traffic moment. In a heavy traffic day that would be a lot of load. And for what? For an instable connection that is not working correctly...
You are right, the plan is to create a secondary server, bypassing cloudflare to allow websockets, and modify the
socketio.addressparameter in config.json to point to that secondary server.
That should work (we hope), until attacked. In case an attacker targets the server by IP, we have to move to another server, turn on cloudflare temporarily and bypass again when the attack finishes. So we still need long polling to keep the forum working under attack. As a fallback option.I'm with @pitaj and @zack, long polling should work.
I'm a bit disappointed, I was pretty sure this was a problem or misconfiguration on our side, but not the expected behavior. You guys did a really good job with NodeBB, I've said that before, and the API first approach is perfect also. Websockets are the future, but DDoS attacks are going to be there too. Long polling support is critical. -
Guys, long polling does work. If your browser or ISP doesn't support the websocket protocol, NodeBB will drop down to xhr-polling, and all functionality should continue working.
Now, the exception is that websockets doesn't work well with CloudFlare, and somehow socket.io cannot realise this and continues to try to upgrade the connection to websocket.
So the approach here is as always:
Use cloudflare as DNS only, without acceleration.
Grey clouds, over orange.
-
Thanks for the answer @julian, but "grey clouds" (or bypass) is the same as removing cloudflare in terms of security: no DDoS protection at all.
We use Cloudflare as a security measure, cause it hides our server behind CloudFlare's IPs, thus removing the option to target our server directly with a DDoS attack.
If we use Cloudflare as DNS only, then everyone will know our IP, and that would be bad hehe.We will keep investigating the issue, hope we can find the reason long polling is not working properly. I'm not sure it is Cloudflare, nginx is also in the middle, and NodeBB in the end, a lot of things we have to check.
-
@TheBronx Please double check with CF, but I believe that is incorrect.
Bypassing CloudFlare via grey clouds just bypasses the acceleration and caching provided by CF. Their DDoS protection works on the DNS level, and you will still be using CloudFlare for that, and they will still be cloaking your IP address.
-
just ping a bypassed domain, and you will see the real IP address, not a cloudflare's one.
an attacker can send traffic to the IP, without even reaching cloudflare servers.
for example, a DNS amplification attack:
Imagine our server is on 1.2.3.4
An attacker sends DNS queries to vulnerable DNS resolvers, spoofing your IP address:- Hello DNS resolver, I'm 1.2.3.4, please, tell me all you know about nasa.gov
- Server 1.2.3.4 here is your information!
Send a few thousand requests like that, and 1.2.3.4 is fucked. CloudFlare? CloudFlare knows nothing about this attack, nobody even asked him anything.
-
The site works with xhr-polling, to my knowledge. We had a client using xhr-polling for awhile, with no loss of functionality...
There are many ajax requests, yes, but that is the nature of long polling, no? Then again there shouldn't be multiple calls per second... just one or two every 10s or so...
-
Some more thoughts on the issue:
- When you don't have websockets working, we drop down to xhr polling (as established prior).
- To establish a secure connection, socket.io goes through a multi-stage handshake to exchange information
- If you are utilising multiple NodeBB processes, separate parts of the handshake can end up going to different processes, and the handshake is lost.
@markkus @TheBronx Can you try reducing the number of ports used by NodeBB to just 1, and see if the issue persists? I have a feeling it will not, though of course, this is not a solution...
-
@julian you are right, when setting only one instance, long polling works as expected. It only uses one request at a time, that gets an answer soon or late, but it no longer uses 4 or 5 request at the same time. there are also no 400 errors.
we have to go back to multiple instances, of course, but it seems you have found something. is there anything we can do to make nginx redirect all requests from the same IP (or user) to the same nodebb instance? this should help, am I wrong?
by the way, we have tried with a secondary server (we still have to fix some issues), and we have found that the wiki is wrong about the socketio configuration:
it says you have to create a block in config.json like:
"socketio": { ... }
when it should say:
"socket.io": { ... }
notice the dot xD -
The solution for this problem (in addition to using
ip_hash, that is) as suggested online seems to be to use thesticky-sessionsmodule, although NodeBB actually doesn't use the cluster module, so this is not an applicable solution unless we rewrite the load balancing code inloader.jsto use Node.js'clustermodule again.
