Posts made by djensen47

@PitaJ said in spawn ENOMEM: Memory leak in 1.19.1?:

Does the memory usage grow quickly at first and then slow down approaching an asymptote?

I believe so.

Today the node processes are at 32% and 30% of memory.
Redis is steady at 2.6% (roughly same as before).

I can keep going but eventually it's just going to keep growing until all the memory is consumed and then it goes to swap and because the memory isn't freeing up, it's not going to help.

It's a slooow leak but it's a leak.

@julian @boris Traffic is normal, probably a bit lower than normal when the incident occurred. Traffic was much higher in March/April 2020 than it is now.

I'll get the memory info but at the moment each process is using 25% of available memory. I rebooted the server about 5 hours ago.

@julian Swap won't solve a memory leak though. In fact, it'll just hide the issue until the server is unmanageably slow, no?

Also, this is new behavior that started sometime between version 1.17 and 1.19 (no issues on 1.16).

I'm replying because we ran into the issue again.

@julian @boris

IIRC, swap is configured by default. I have two processes, 1 per CPU.

MongoDB is on a separate server. Although, it looks like Redis is on the same server. Maybe Redis isn’t cleaning up old sessions, again?

A few users received this error message while trying to login:

spawn ENOMEM

1. Why is login trying to spawn a process?
2. Memory was maxed out when I checked. I'm running with 2GB.

It looks like the memory leak has been here longer:
Jan. 24 is when I upgraded from 1.16 to 1.19

I've never had memory issues with NodeBB

@pitaj Awesome, thank you!

I was running NodeBB 16.x and recently updated to 17.x → 18.x → 19.x and somewhere along the way the API that TripleA (board game software that posts to my forum) uses had a breaking change. Looking through the changelog, I can't figure out which change it was.

For reference here's the issue on TripleA. The stack trace may not be useful here but just in case..

https://github.com/triplea-game/triplea/issues/10032

tripleabuilderbot created this issue in triplea-game/triplea

closed 2.5.22294: NodeBbForumPoster#uploadSaveGame:163 - java.lang.ClassCastException #10032

I recently updated the dice-bot plugin to 1.1.0

Somehow it was uninstalled. Weird.
There was also a bug in the code. Maybe a node.js version issue.

I think it's something else but I am seeing this error in the logs:

error: Error: ENOENT: no such file or directory, open '/var/srv/nodebb/node_modules/nodebb-plugin-require/plugin.json'   

@pitaj said in Breaking change for plugins:

Which probably means spinning up a dev instance and going through each version.

Honestly, this sounds like a waste of time.

I'll just figure it out myself. Thanks anyway.

@pitaj No errors. No, I cannot identify which major version broke it.

I migrated one version at a time but all in one sitting.

I combed through the release notes and I likely missed the breaking change to plugins. But I can't find the change that would've broken my plugin here: https://github.com/djensen47/nodebb-plugin-dice-bot

It worked on 1.12 and breaks in 1.16

Thanks in advance!

@julian Yeah, I might have somebody in mind.

I did a massive migration 2 years ago and as a result nearly everybody in the system was set to "weekly" as the default digest setting. I also have emails the as so old they will bounce.

1. What query/script could I run to change all the weekly values to system default?
2. Given a list of email addresses, what query or script could I run to change the digest value to none or whatever "off" would be.

Thanks in advance.

Since this is related to bugs, I thought I would post it here (and not Github).

I'm not sure what the NodeBB team does for regression testing but forum-in-subdirectory seems to break something every few releases. If you have a test suite or regression testing setup, I would suggest a copy of that for forum-in-subdirectory.

Thanks for listening!

@baris Yeah my browser tab icon looks really chunky right now.

Aside: Stay tuned for one and possibly numerous bugs related to my forum being in a subdirectory.

A reboot seemed to help. Though, I'm still not sure why it shows the 16x16 icon instead of the nice Progressive Web icon