socket.io log flooding



  • This is part of my log

    io: 0 emit [ 'disconnecting', 'ping timeout' ]
    io: 0 emit [ 'disconnect', 'ping timeout' ]
    io: 0 emit [ 'disconnecting', 'ping timeout' ]
    io: 0 emit [ 'disconnect', 'ping timeout' ]
    io: 0 emit [ 'disconnecting', 'transport close' ]
    io: 0 emit [ 'disconnect', 'transport close' ]
    io: 0 emit [ 'disconnecting', 'ping timeout' ]
    io: 0 emit [ 'disconnect', 'ping timeout' ]
    io: 0 emit [ 'disconnecting', 'transport close' ]
    io: 0 emit [ 'disconnect', 'transport close' ]
    io: 0 emit [ 'disconnecting', 'transport close' ]
    io: 0 emit [ 'disconnect', 'transport close' ]
    io: 0 emit [ 'disconnecting', 'ping timeout' ]
    io: 0 emit [ 'disconnect', 'ping timeout' ]
    io: 0 emit [ 'disconnecting', 'transport close' ]
    

    Only a few logs are normal requests, and the node process eat up most of the CPU resources.

    UPDATE 2016/11/3:

    "socket.io": {
        "transports": ["websocket"]
    }
    

    The performance seems better after I applied this change in config.

    Behaviors before change:

    • Most of the logs are io emit timeout or io emit transport close . Only very rare portion (less than 10 percent) of requests successfully processed.
    • Every time I restart the site it can not response for about 5 minutes or longer even I see in the log that process has already listened to 4567.
    • Some of the socket.io/EIO?xxxx requests failed at client side.

    Behaviors after change:

    • No io emit timeout was found ever
      io emit transport close still exists, about 50% or less at peak hour and 10% at valley hour.
      *The site is fast enough at peak hour and it can recover quickly even I restart.
      *None of the requests failed at client side.

  • Global Moderator

    are you using cloudflare?

    looks like your sockets aren't working right. Can you show client-side logs too?



  • @PitaJ
    Thanks for the hint.
    The client side log is described here
    https://community.nodebb.org/topic/9752/some-polling-requests-fail



  • @PitaJ
    We run it on linode machine, not cloudflare.

    By the way, we had run it on Digital Ocean machine, it also behave like this.


  • Global Moderator

    Yeah it looks like it's failing to connect directly to websockets and is failing back on long polling.



  • @PitaJ

    Can we disable socket?


  • Global Moderator

    @xidui no you need sockets for NodeBB to work right.



  • I notice that when the site is at peak hour, most of the logs are io logs such as disconnecting or disconnect.
    When not during the peak hours, such error becomes less but still exists.

    Seems the nginx config is right, other wise socket can not work even during valley hours.

    @administrators
    I wonder is their any limit of user in NodeBB. During peak hour, users active exceeds 1000 monitors at admin page, is that size too large for Nodebb to hold in on single machine?



  • I added this config:

    "socket.io": {
        "transports": ["websocket"]
    }
    

    The flooding error seems less than before. But still need to monitor over several days.


  • Admin

    If you have more than 1000 users online on a single process you might be hitting open file limits on your OS try increasing those. https://easyengine.io/tutorials/linux/increase-open-files-limit/



  • @baris

    That's single machine, 4 processes and the 1000 is active user not online user.



  • @xidui said in socket.io log flooding:

    I added this config:

    "socket.io": {
        "transports": ["websocket"]
    }
    

    The flooding error seems less than before. But still need to monitor over several days.

    The performance seems better after I applied this change in config.

    1. Behaviors before change:
      • Most of the logs are io emit timeout or io emit transport close . Only very rare portion (less than 10 percent) of requests successfully processed.
      • Every time I restart the site it can not response for about 5 minutes or longer even I see in the log that process has already listened to 4567.
      • Some of the socket.io/EIO?xxxx requests failed at client side.
    1. Behaviors after change:
      • No io emit timeout was found ever
      • io emit transport close still exists, about 50% or less at peak hour and 10% at valley hour.
      • The site is fast enough at peak hour and it can recover quickly even I restart.
      • None of the requests failed at client side.

    @administrators
    I think my issue has accidentally and surprisingly solved by this small change temporarily. I wonder why and is this config recommended or ["polling", "websocket"] is better? And do you have some idea on the root cause of the issue when I use ["polling", "websocket"]?



  • Thanks for every one!

    This issue seems to disappear after the change in config. Our site is more stable than ever before in the last week with a rather short response time during the whole week.

    Thanks again and welcome to have a visit. Advices on the site is always welcomed.


Log in to reply
 

Looks like your connection to NodeBB was lost, please wait while we try to reconnect.