Debugging nodejs app.js at 100%

exodo

I have been facing a problem where some of the cores gets stuck with nodejs at 100% for hours and hours while not many online users or during the night

I´m looking for some info to debugg which commands are making this problem

i have found this but not sure how to test it with nodebb

You can profile your app with node-tick.

Install node-tick by sudo npm -g install tick
Run your app with enabled profile node --prof ./app.js
After some time with CPU 100% usage stop your app
You can see v8.log in your app directory, now you can read it with node-tick-processor
Run node-tick-processor and explain results
Load v8.log into chrome://tracing to analyse as tree.

any helop will be apreciated

exodo

Please move it to technical support maybe

hukadan

I cannot help you with Node itself. But if I may, you should run NodeBB using an unprivileged user. You will save yourself a lot of trouble. May be you should also change the user running nginx from nobody to something more sensible like www or www-data.

Also, the Node and NodeBB version would certainly help Node gurus.

<baris>

Post nodejs version disable all custom plugins and see if it still happens. Try updating to nodejs 4.5 and see if it happens.

We are not seeing this issue on this forum or the forums we host. So it's either a nodejs version issue or a custom plugin doing something that's locking the node process. Also check this out https://community.nodebb.org/topic/8998/nodebb-periodically-hangs and https://community.nodebb.org/topic/8789/infinite-loop

exodo

@baris said in Debugging nodejs app.js at 100%:

Post nodejs version disable all custom plugins and see if it still happens. Try updating to nodejs 4.5 and see if it happens.

We are not seeing this issue on this forum or the forums we host. So it's either a nodejs version issue or a custom plugin doing something that's locking the node process. Also check this out https://community.nodebb.org/topic/8998/nodebb-periodically-hangs and https://community.nodebb.org/topic/8789/infinite-loop

my node is 4.4.7, i tried to disable some plugins and see if it makes a difference
thansk for the links

right now stopping nodebb keeps the cores at 100% and i have to kill manually

root@w1-exodo:/var/foroexodo/nodebb# ./nodebb stop
Stopping NodeBB. Goodbye!
root@w1-exodo:/var/foroexodo/nodebb# htop
root@w1-exodo:/var/foroexodo/nodebb# ps aux | grep nodebb
root      4970  0.0  0.0   8100   948 pts/1    S+   17:10   0:00 grep --color=auto nodebb
root     18095  0.0  1.0 971188 65068 ?        Ssl  09:27   0:00 /usr/bin/nodejs /var/foroexodo/nodebb/loader.js
root@w1-exodo:/var/foroexodo/nodebb# kill -9 18095
root@w1-exodo:/var/foroexodo/nodebb# ps aux | grep nodebb
root      5425  0.0  0.0   8100   948 pts/1    S+   17:10   0:00 grep --color=auto nodebb
root@w1-exodo:/var/foroexodo/nodebb# htop
root@w1-exodo:/var/foroexodo/nodebb# ps aux | grep node
root      5434  0.0  0.0   8100   948 pts/1    S+   17:10   0:00 grep --color=auto node
root      5791  0.0  0.0  52136  5444 ?        Ss   Aug29   0:55 /usr/sbin/munin-node
root     18109 36.4 11.0 1861016 673192 ?      Rl   09:27 168:42 /usr/bin/nodejs app.js
root     18114 77.3 12.3 1917928 756612 ?      Rl   09:27 358:27 /usr/bin/nodejs app.js
root@w1-exodo:/var/foroexodo/nodebb# kill -9 18109
root@w1-exodo:/var/foroexodo/nodebb# ps aux | grep node
root      5437  0.0  0.0   8100   948 pts/1    S+   17:10   0:00 grep --color=auto node
root      5791  0.0  0.0  52136  5444 ?        Ss   Aug29   0:55 /usr/sbin/munin-node
root     18114 77.4 12.3 1917928 756612 ?      Rl   09:27 358:40 /usr/bin/nodejs app.js
root@w1-exodo:/var/foroexodo/nodebb# kill -9 18114
root@w1-exodo:/var/foroexodo/nodebb# ps aux | grep node
root      5439  0.0  0.0   8100   944 pts/1    S+   17:10   0:00 grep --color=auto node
root      5791  0.0  0.0  52136  5444 ?        Ss   Aug29   0:55 /usr/sbin/munin-node

exodo

@hukadan said in Debugging nodejs app.js at 100%:

I cannot help you with Node itself. But if I may, you should run NodeBB using an unprivileged user. You will save yourself a lot of trouble. May be you should also change the user running nginx from nobody to something more sensible like www or www-data.

Also, the Node and NodeBB version would certainly help Node gurus.

Thanks for the tips

sloosecannon

@baris said in Debugging nodejs app.js at 100%:

Post nodejs version disable all custom plugins and see if it still happens. Try updating to nodejs 4.5 and see if it happens.

We are not seeing this issue on this forum or the forums we host. So it's either a nodejs version issue or a custom plugin doing something that's locking the node process. Also check this out https://community.nodebb.org/topic/8998/nodebb-periodically-hangs and https://community.nodebb.org/topic/8789/infinite-loop

This happens at wtdwtf too - we have a script that kills an instance if it pegs a core and stops responding IIRC. I believe it's a nodejs bug.

ETA: The reason we suspect it's a nodejs bug, IIRC (it's been a while since we discussed it), is because it's happening in native code at low levels. I think it was suggested that specific virtualized environments are also related to the issue - DigitalOcean is the one we're using.

exodo

@sloosecannon said in Debugging nodejs app.js at 100%:

@baris said in Debugging nodejs app.js at 100%:

Post nodejs version disable all custom plugins and see if it still happens. Try updating to nodejs 4.5 and see if it happens.

We are not seeing this issue on this forum or the forums we host. So it's either a nodejs version issue or a custom plugin doing something that's locking the node process. Also check this out https://community.nodebb.org/topic/8998/nodebb-periodically-hangs and https://community.nodebb.org/topic/8789/infinite-loop

This happens at wtdwtf too - we have a script that kills an instance if it pegs a core and stops responding IIRC. I believe it's a nodejs bug.

ETA: The reason we suspect it's a nodejs bug, IIRC (it's been a while since we discussed it), is because it's happening in native code at low levels. I think it was suggested that specific virtualized environments are also related to the issue - DigitalOcean is the one we're using.

Weird
Thanls for the info
We should work all togetjer to try solve this
Can you share that script?

jarey

If the problem doesn't replicate on this community maybe using from now the same version and builf of nodejs used on this community might help in order to not suffer the problem. Anyone knows what version is running here?

Thanks.

<baris>

@jarey We are using nodejs 4.5.0 here.

Bri

I haven't seen problems like this either, running on node v4.3.2

julian

It may be related to traffic and heavy usage though... this community isn't as heavily used as some other ones.

However that said, we do host some forums with heavy traffic and don't see issues like this, though they are spread out over multiple cores/servers so each individual NodeBB server handles a fraction of the load.

exodo

@julian said in Debugging nodejs app.js at 100%:

It may be related to traffic and heavy usage though... this community isn't as heavily used as some other ones.

However that said, we do host some forums with heavy traffic and don't see issues like this, though they are spread out over multiple cores/servers so each individual NodeBB server handles a fraction of the load.

Yes. You wont see this issue on low load communities probably, i dont call mine a heavy traffic anyways.
Our server is not phisical and it can handle much more traffic, 12 cores.
The problem must be some function that gets random core at 100% and stuck. That ends on some instances not working and nodebb stop doing nothing at all.

sloosecannon

@exodo said in Debugging nodejs app.js at 100%:

@sloosecannon said in Debugging nodejs app.js at 100%:

@baris said in Debugging nodejs app.js at 100%:

Post nodejs version disable all custom plugins and see if it still happens. Try updating to nodejs 4.5 and see if it happens.

We are not seeing this issue on this forum or the forums we host. So it's either a nodejs version issue or a custom plugin doing something that's locking the node process. Also check this out https://community.nodebb.org/topic/8998/nodebb-periodically-hangs and https://community.nodebb.org/topic/8789/infinite-loop

This happens at wtdwtf too - we have a script that kills an instance if it pegs a core and stops responding IIRC. I believe it's a nodejs bug.

ETA: The reason we suspect it's a nodejs bug, IIRC (it's been a while since we discussed it), is because it's happening in native code at low levels. I think it was suggested that specific virtualized environments are also related to the issue - DigitalOcean is the one we're using.

Weird
Thanls for the info
We should work all togetjer to try solve this
Can you share that script?

I don't have access to it, but I'll mention the server admins and see if they can help.

@Ben-Lubar, @boomzilla - can either of you post the restart-script we use at WTDWTF?

Ben Lubar

@exodo @sloosecannon It's this:

https://github.com/boomzillawtf/tdwtf/blob/master/plugins/nodebb-plugin-tdwtf-customizations/index.js#L13
https://github.com/boomzillawtf/tdwtf/blob/master/watchdog.bash

exodo

@Ben-Lubar said in Debugging nodejs app.js at 100%:

@exodo @sloosecannon It's this:

https://github.com/boomzillawtf/tdwtf/blob/master/plugins/nodebb-plugin-tdwtf-customizations/index.js#L13
https://github.com/boomzillawtf/tdwtf/blob/master/watchdog.bash

Thank you

exodo

@baris said in Debugging nodejs app.js at 100%:

@jarey We are using nodejs 4.5.0 here.

Just installed 4.5.0
Issue is still there

exodo

Going back to the thread purpose.
Is it possible to run nodebb with node-tick enabled?

PitaJ

@exodo I would imagine it's something like node --prof app.js