spawn ENOMEM: Memory leak in 1.19.1?

Dave Jensen

IIRC, swap is configured by default. I have two processes, 1 per CPU.

MongoDB is on a separate server. Although, it looks like Redis is on the same server. Maybe Redis isn’t cleaning up old sessions, again?

Dave Jensen

@julian Swap won't solve a memory leak though. In fact, it'll just hide the issue until the server is unmanageably slow, no?

Also, this is new behavior that started sometime between version 1.17 and 1.19 (no issues on 1.16).

I'm replying because we ran into the issue again.

<baris>

What does top show? How much traffic is there when it goes down.

julian

@baris said in spawn ENOMEM: Memory leak in 1.19.1?:

What does top show?

Yeah, would be good to know how much memory your NodeBB is consuming at that point in time, vs. freshly started.

Dave Jensen

@julian @boris Traffic is normal, probably a bit lower than normal when the incident occurred. Traffic was much higher in March/April 2020 than it is now.

I'll get the memory info but at the moment each process is using 25% of available memory. I rebooted the server about 5 hours ago.

Dave Jensen

Today the node processes are at 32% and 30% of memory.
Redis is steady at 2.6% (roughly same as before).

I can keep going but eventually it's just going to keep growing until all the memory is consumed and then it goes to swap and because the memory isn't freeing up, it's not going to help.

It's a slooow leak but it's a leak.

PitaJ

@djensen47 maybe it's the node-side db cache. I'm not sure what kind of memory limits we set for it. Does the memory usage grow quickly at first and then slow down approaching an asymptote?

<baris>

Please share output of top sorted by memory usage and free -m

Dave Jensen

Dave Jensen

@PitaJ said in spawn ENOMEM: Memory leak in 1.19.1?:

Does the memory usage grow quickly at first and then slow down approaching an asymptote?

I believe so.