Unsolved spawn ENOMEM: Memory leak in 1.19.1?

[djensen47]

A few users received this error message while trying to login:

spawn ENOMEM

1. Why is login trying to spawn a process?
2. Memory was maxed out when I checked. I'm running with 2GB.

It looks like the memory leak has been here longer:
Jan. 24 is when I upgraded from 1.16 to 1.19

I've never had memory issues with NodeBB

[baris]

Login spawns a process for the password hashing so it doesn't block the main thread.

Does your system have swap configured? How many nodebb processes are being used?

[julian]

@djensen47 Also for what it's worth, MongoDB also tends to allocate as much memory as is free, for its own use, so it might not actually be NodeBB (although it doesn't hurt to check)

It doesn't usually cause issues with running out of memory, but occasionally can. This is warded off with swapfile usage.

[djensen47]

@julian @boris

IIRC, swap is configured by default. I have two processes, 1 per CPU.

MongoDB is on a separate server. Although, it looks like Redis is on the same server. Maybe Redis isn’t cleaning up old sessions, again?

[djensen47]

@julian Swap won't solve a memory leak though. In fact, it'll just hide the issue until the server is unmanageably slow, no?

Also, this is new behavior that started sometime between version 1.17 and 1.19 (no issues on 1.16).

I'm replying because we ran into the issue again.

[baris]

What does top show? How much traffic is there when it goes down.

[julian]

@baris said in spawn ENOMEM: Memory leak in 1.19.1?:

What does top show?

Yeah, would be good to know how much memory your NodeBB is consuming at that point in time, vs. freshly started.

[djensen47]

@julian @boris Traffic is normal, probably a bit lower than normal when the incident occurred. Traffic was much higher in March/April 2020 than it is now.

I'll get the memory info but at the moment each process is using 25% of available memory. I rebooted the server about 5 hours ago.

[djensen47]

Today the node processes are at 32% and 30% of memory.
Redis is steady at 2.6% (roughly same as before).

I can keep going but eventually it's just going to keep growing until all the memory is consumed and then it goes to swap and because the memory isn't freeing up, it's not going to help.

It's a slooow leak but it's a leak.

[PitaJ]

@djensen47 maybe it's the node-side db cache. I'm not sure what kind of memory limits we set for it. Does the memory usage grow quickly at first and then slow down approaching an asymptote?

[baris]

Please share output of top sorted by memory usage and free -m

[djensen47]

[djensen47]

@PitaJ said in spawn ENOMEM: Memory leak in 1.19.1?:

Does the memory usage grow quickly at first and then slow down approaching an asymptote?

I believe so.