NodeBB 1.11.2 Error Connect EMFILE

scottalanmiller

We've been getting this quite a bit recently. We are on 1.11.2. We get scores of these errors as the site gets flaky and eventually crashes completely. A restart bring it back up fine and it runs for a while without an issue. Been happening for a few weeks now.

2019-03-01T17:06:04.467Z [4567/4257] - error: connect EMFILE 35.172.4.119:465 - Local (undefined:undefined)
Error: connect EMFILE 35.172.4.119:465 - Local (undefined:undefined)
    at internalConnect (net.js:932:16)
    at defaultTriggerAsyncIdScope (internal/async_hooks.js:294:19)
    at defaultTriggerAsyncIdScope (net.js:1024:9)
    at process._tickCallback (internal/process/next_tick.js:61:11) {"errno":"EMFILE","code":"ECONNECTION","syscall":"connect","address":"35.172.4.119","port":465,"command":"CONN"}

julian

@scottalanmiller @baris, too many open files?

Have you tried raising ulimit?

scottalanmiller

No, you think that we might be short on file handles due to volume?

scottalanmiller

@julian said in NodeBB 1.11.2 Error Connect EMFILE:

@scottalanmiller @baris, too many open files?

Have you tried raising ulimit?

Raised it now. We will see...

scottalanmiller

@julian said in NodeBB 1.11.2 Error Connect EMFILE:

@scottalanmiller @baris, too many open files?

Have you tried raising ulimit?

Raised it dramatically. Seemed to help for a little while. But also updated to 1.12.0 so that might have had an effect.

But now getting this...

2019-03-13T15:42:05.728Z [4567/31594] - error: connect EMFILE 52.10.168.253:465 - Local (undefined:undefined) {"errno":"EMFILE","code":"ECONNECTION","syscall":"connect","address":"52.10.168.253","port":465,"command":"CONN"}
2019-03-13T15:42:05.728Z [4567/31594] - error: connect EMFILE 52.10.168.253:465 - Local (undefined:undefined) {"errno":"EMFILE","code":"ECONNECTION","syscall":"connect","address":"52.10.168.253","port":465,"command":"CONN"}
2019-03-13T15:42:05.728Z [4567/31594] - error: connect EMFILE 52.10.168.253:465 - Local (undefined:undefined) {"errno":"EMFILE","code":"ECONNECTION","syscall":"connect","address":"52.10.168.253","port":465,"command":"CONN"}

PitaJ

Could you check the open file descriptors with lsof -p <PID> periodically? My guess is that you're running into limitations because you have so many open websockets. Perhaps running more NodeBB processes would help.

scottalanmiller

Wait, I think the shell was holding onto the old ulimit. Testing again.

julian

@scottalanmiller Fluentd hosts some nice configs (distilled from a presentation from a Netflix engineer about tuning their servers) https://docs.fluentd.org/v1.0/articles/before-install

Perhaps this could help? Also check out ss -s, what is TIMEWAIT?