NodeBB periodically hangs

Mikael Jenkler

Cool NVM, something learned every day Good to know for dev sites!

L33t

Bad news is that I crashed with absolutely no errors a few minutes ago. The good news is that it actually recovered on it's on in dev mode. I'll try updating nodejs as recommended.

L33t

Looks like moving to the latest node.js did not help. Just happened again.

boomzilla

We're running our instance with --perf-basic-prof-only-functions now and are able to generate flamegraphs. Just waiting for the servercooties to strike now...

Example flamegraph:
nodebb flamegraph

jiangcaiyang

Never heard of it...
I am a lab rat...

boomzilla

Here's a thing. Had a nodebb process pegging the CPU and ran the perf tool against it. It collected 2,967 identical stacks:


node 29869 cpu-clock: 
            3288d1761413 LazyCompile:~ /usr/src/app/node_modules/nodebb-plugin-mentions/library.js:165 (/tmp/perf-29869.map)
            3288d1772468 LazyCompile:~ /usr/src/app/node_modules/nodebb-plugin-mentions/library.js:239 (/tmp/perf-29869.map)
            3288d1770e25 LazyCompile:~ /usr/src/app/node_modules/nodebb-plugin-mentions/library.js:209 (/tmp/perf-29869.map)
            3288d135f1f9 LazyCompile:~ /usr/src/app/src/database/mongo/sorted.js:341 (/tmp/perf-29869.map)
            3288d179b3fa LazyCompile:* /usr/src/app/src/topics/unread.js:340 (/tmp/perf-29869.map)
            3288d15bd907 LazyCompile:*readableAddChunk _stream_readable.js:124 (/tmp/perf-29869.map)
            3288d13e12ce LazyCompile:*Readable.push _stream_readable.js:100 (/tmp/perf-29869.map)
            7f52c6fdbb45 init_cacheinfo (/lib/x86_64-linux-gnu/libc-2.19.so)

So now the mentions plugin is suspect #1. cc: @julianlam

pichalite

@boomzilla anything on the database resource usage stats at that point in time?

<baris>

@boomzilla maybe disable the plugin and let it run and see if you have the same problem. Do you guys get this hanging issue daily?

Could be related to https://github.com/NodeBB/NodeBB/issues/4395

Ben Lubar

@baris said in NodeBB periodically hangs:

Do you guys get this hanging issue daily?

We've been tracking the manual restarts here: https://what.thedailywtf.com/topic/19990/the-official-server-cooties-tracking-thread

Ben Lubar

Is anyone who is affected by this using the dbsearch plugin with a MongoDB database? I got rid of ours and the periodic hangs went away, at least for the last 24 hours.

PitaJ

@Ben-Lubar hmmm. If you have a lot of posts and dbsearch has a synchronous loop this could be the cause.

AOKP

To be honest I think that the actual issue here is a plugin or a poor setup overall, if not even a custom javascript interefering with one of NodeBB's internals. As of now I had no longs from 0.7.x and up.

How about adding additional info:

NodeJS version
Cluster?
How do you start NodeBB?