warn: Flooding detected! .... warn: [socket.io] Too many emits!

[j0hn0ah]

We are running nodebb 1.10.1 on a fairly busy server, configured currently with 8 node processes (which nginx is proxying across), and everything seems to work pretty well as far as the forum's web UI responsiveness / lack of 503s / etc (these are some issues we ran into earlier, before we had properly configured it to use more than one node process).

Recently I was checking the nodebb logs and I came across a number of messages like the following -

$ grep -A1 "Flooding" output.log | head -2
2018-11-10T19:43:36.348Z [7599] - warn: Flooding detected! Calls : 101, Duration : 2
2018-11-10T19:43:36.352Z [7599] - warn: [socket.io] Too many emits! Disconnecting uid : 47371. Events : plugins.composer.renderPreview,plugins.composer.renderPreview,plugins.composer.renderPreview,plugins.composer.renderPreview,plugins.composer.renderPreview,plugins.composer.renderPreview,plugins.composer.renderPreview,plugins.composer.renderPreview,plugins.composer.renderPreview,plugins.composer.renderPreview,plugins.composer.renderPreview,plugins.composer.renderPreview,plugins.composer.renderPreview,plugins.composer.renderPreview,plugins.composer.renderPreview,plugins.composer.renderPreview,plugins.composer.renderPreview,plugins.composer.renderPreview,plugins.composer.renderPreview,plugins.composer.renderPreview

$ grep -A1 "Flooding" output.log | tail -2
2018-12-14T17:48:42.266Z [2325] - warn: Flooding detected! Calls : 101, Duration : 7459
2018-12-14T17:48:42.268Z [2325] - warn: [socket.io] Too many emits! Disconnecting uid : 0. Events : categories.loadMore,categories.loadMore,categories.loadMore,categories.loadMore,categories.loadMore,categories.loadMore,categories.loadMore,categories.loadMore,categories.loadMore,categories.loadMore,categories.loadMore,categories.loadMore,categories.loadMore,categories.loadMore,categories.loadMore,categories.loadMore,categories.loadMore,categories.loadMore,categories.loadMore,categories.loadMore

$ grep -c "Flooding detected" output.log
272

I haven't been able to to 100% correlate this with actual performance issues on the forum, but it's still a bit worrying.. we have had occasional complaints about responsiveness, and actually we had significant problems before a recent change from 4 nodebb ports/processes to 8, which helped immensely.

In any case, from the research I've done this is related to socket.emit() firing too often (20 times a second is the number I keep seeing)... If you count up the number of comma separated print outs of "categories.loadMore", or plugins.composer.renderPreview in the first example, in each of those log lines, it is exactly 20.

Is this a genuine issue to worry about? Is it possible this is just because the forum is doing so much traffic? Does anyone have any suggestions on how to drill down farther into this?

thanks!
John

[baris]

Those messages are printed when a single socket(user) sends a high number of socket.emits in a short duration. You can find the relevant code here https://github.com/NodeBB/NodeBB/blob/master/src/middleware/ratelimit.js#L10. If a user makes more than 100 socket.emits in the span of 10 seconds then they are disconnected.

As for why you are seeing them my guess is those users had connection problems and when they got reconnected socket.io sent a lot of emits instantly which triggered the flooding logic.

[j0hn0ah]

@baris said in warn: Flooding detected! .... warn: [socket.io] Too many emits!:

Those messages are printed when a single socket(user) sends a high number of socket.emits in a short duration. You can find the relevant code here https://github.com/NodeBB/NodeBB/blob/master/src/middleware/ratelimit.js#L10. If a user makes more than 100 socket.emits in the span of 10 seconds then they are disconnected.
As for why you are seeing them my guess is those users had connection problems and when they got reconnected socket.io sent a lot of emits instantly which triggered the flooding logic.

Thank you, this is helpful!

Is it possible that this is legit traffic from a user, would a browser and nodebb ever be that chatty? Or do you think it is more likely someone probing / poking at the server in some way? Thanks!

[j0hn0ah]

Sorry, I didn't really process your full message. At the end you state your guess these are users with connection problems who got reconnected... based on the relatively small number of times I see this warning, as compared to over-all traffic to the forum, it sounds like I can probably ignore the warning. Thanks for your help.