NodeBB periodically hangs

pichalite

@L33t git rev-parse HEAD

L33t

@pichalite That'd work if I didn't just scp it to another server.

yariplus

Try grep -c filter:hotswap src/webserver.js

if it says 0, you are on stable.

boomzilla

We see similar issues. Sometimes we have to restart many times per day, some days it will keep going. we've been tracking it here:

https://what.thedailywtf.com/topic/19990/the-official-server-cooties-tracking-thread

We're running stuff under docker, which makes things more complicated, but I've had luck in a local dev/test environment with using Linux perf tools and building flamegraphs. For raisins, we haven't done this yet in docker, though you can see how to make that work here.

However, if you're not running docker, it's a little bit easier. Follow the instructions here:

https://yunong.io/2015/11/23/generating-node-js-flame-graphs/

He doesn't go into it there, but if you're on Ubuntu you'll need to install:

linux-tools-common
linux-cloud-tools-common

...and then kernel specific versions of those based on what you're running in order to use perf.

If you get there before we do, I'm eager to hear what anyone discovers with this.

L33t

@yariplus I'm on stable.

@boomzilla Thanks for the info. I'm on CentOS and not using docker. I'll take a look at that sometime soon. It's late thought and it's been a long day.

LukeLaupheimer

This is pretty hacky but it's worked beautifully so far: I wrote a Ruby script that tries to load my page. If it fails, it restarts the server. This is run in a cron job every minute. Here's my code:

require 'net/http'

begin
	print "Downloading the page..."
	res = Net::HTTP.start("yourdomainhere.com", :read_timeout => 5)
	res.get('/')
	puts " success!"
rescue Net::ReadTimeout => e
	print " failed.\n\nRestarting the server..."
	`./nodebb restart`

	puts " success!"
end

Until the cause of this is figured out, this or something like it might be a good enough workaround. It sure beats me manually restarting it after my users throw a fit.

L33t

@LukeLaupheimer That's beautifully terrible. Once it crashes again and I (maybe) get some useful info.. I'll do something like that.

I don't ruby. Is that a loop? Or do I need to set a cron to run it every x minutes? Also.. my http is a 301 to https. Will that work with https also?

LukeLaupheimer

@L33t I set up a cron job so that it wouldn't be subject to the server resetting. I wanted it to be somewhat resilient in spite of its hackiness.

That said, this is just treating a symptom. I absolutely do not advocate for this approach on a permanent basis, only until the true cause has been revealed and addressed. It's the software engineers' equivalent of filling a hole in a dam with a big wad of gum. It buys you time. It doesn't solve the real issue.

Mikael Jenkler

In my case the issue was with an old version of Nodejs. Try to update it.

ranux

After update to latest node.js version, everything works fine. I recommend use nvm for switching between node.js versions.

Mikael Jenkler

Cool NVM, something learned every day Good to know for dev sites!

L33t

Bad news is that I crashed with absolutely no errors a few minutes ago. The good news is that it actually recovered on it's on in dev mode. I'll try updating nodejs as recommended.

L33t

Looks like moving to the latest node.js did not help. Just happened again.

boomzilla

We're running our instance with --perf-basic-prof-only-functions now and are able to generate flamegraphs. Just waiting for the servercooties to strike now...

Example flamegraph:
nodebb flamegraph

jiangcaiyang

Never heard of it...
I am a lab rat...

boomzilla

Here's a thing. Had a nodebb process pegging the CPU and ran the perf tool against it. It collected 2,967 identical stacks:


node 29869 cpu-clock: 
            3288d1761413 LazyCompile:~ /usr/src/app/node_modules/nodebb-plugin-mentions/library.js:165 (/tmp/perf-29869.map)
            3288d1772468 LazyCompile:~ /usr/src/app/node_modules/nodebb-plugin-mentions/library.js:239 (/tmp/perf-29869.map)
            3288d1770e25 LazyCompile:~ /usr/src/app/node_modules/nodebb-plugin-mentions/library.js:209 (/tmp/perf-29869.map)
            3288d135f1f9 LazyCompile:~ /usr/src/app/src/database/mongo/sorted.js:341 (/tmp/perf-29869.map)
            3288d179b3fa LazyCompile:* /usr/src/app/src/topics/unread.js:340 (/tmp/perf-29869.map)
            3288d15bd907 LazyCompile:*readableAddChunk _stream_readable.js:124 (/tmp/perf-29869.map)
            3288d13e12ce LazyCompile:*Readable.push _stream_readable.js:100 (/tmp/perf-29869.map)
            7f52c6fdbb45 init_cacheinfo (/lib/x86_64-linux-gnu/libc-2.19.so)

So now the mentions plugin is suspect #1. cc: @julianlam

pichalite

@boomzilla anything on the database resource usage stats at that point in time?

<baris>

@boomzilla maybe disable the plugin and let it run and see if you have the same problem. Do you guys get this hanging issue daily?

Could be related to https://github.com/NodeBB/NodeBB/issues/4395

Ben Lubar

@baris said in NodeBB periodically hangs:

Do you guys get this hanging issue daily?

We've been tracking the manual restarts here: https://what.thedailywtf.com/topic/19990/the-official-server-cooties-tracking-thread

Ben Lubar

Is anyone who is affected by this using the dbsearch plugin with a MongoDB database? I got rid of ours and the periodic hangs went away, at least for the last 24 hours.