NodeBB won't restart

nsuchy

After upgrading to NodeBB 1.6.1 I am unable to start the service.

./nodebb dev output:

root@vps146875:/var/www/sinfulforumsnet/NodeBB# ./nodebb dev

NodeBB v1.6.1 Copyright (C) 2013-2014 NodeBB Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it under certain conditions.
For the full license, please visit: http://www.gnu.org/copyleft/gpl.html

Clustering enabled: Spinning up 1 process(es).

16/10 09:22:46 [1783] - verbose: * using configuration stored in: /var/www/sinfulforumsnet/NodeBB/config.json
16/10 09:22:46 [1783] - info: Initializing NodeBB v1.6.1 https://sinfulforums.net:443
16/10 09:22:46 [1783] - verbose: * using mongo store at 127.0.0.1:27017
16/10 09:22:46 [1783] - verbose: * using themes stored in: /var/www/sinfulforumsnet/NodeBB/node_modules
16/10 09:22:46 [1783] - warn: You have no mongo username/password setup!
16/10 09:22:50 [1783] - verbose: [minifier] utilizing a maximum of 0 additional threads
16/10 09:22:50 [1783] - verbose: Checking dependencies for outdated modules
16/10 09:22:51 [1783] - verbose: [notifications.init] Registering jobs.
16/10 09:22:51 [1783] - verbose: [user/jobs] (Re-)starting user jobs...
16/10 09:22:51 [1783] - verbose: [user/jobs] Starting job (digest.daily)
16/10 09:22:51 [1783] - verbose: [user/jobs] Starting job (digest.weekly)
16/10 09:22:51 [1783] - verbose: [user/jobs] Starting job (digest.monthly)
16/10 09:22:51 [1783] - verbose: [user/jobs] Starting job (reset.clean)
16/10 09:22:51 [1783] - verbose: [user/jobs] 4 jobs started
16/10 09:22:51 [1783] - verbose: [emailer] Setting up SMTP fallback transport
16/10 09:22:51 [1783] - info: [themes] Using theme nodebb-theme-persona
16/10 09:22:51 [1783] - verbose: [plugins] Initializing plugins system
16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-composer-default
16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-theme-persona
16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-markdown
16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-mentions
16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-widget-essentials
16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-rewards-essentials
16/10 09:22:51 [1783] - verbose: [plugins] Found 3 sound file(s) for plugin nodebb-plugin-soundpack-default
16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-soundpack-default
16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-emoji-extended
16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-emoji-one
16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-2factor
16/10 09:22:51 [1783] - warn: [plugins/nodebb-plugin-spam-be-gone] Hook `action:flag.create` is deprecated, please use `action:flags.create` instead.
16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-spam-be-gone
16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-emailer-smtp
16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-dbsearch
16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-emailer-sendgrid
16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-imgur
16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-desktop-notifications
16/10 09:22:51 [1783] - info: [plugins] Plugins OK
16/10 09:22:52 [1783] - info: [plugins/spam-be-gone] Settings loaded
16/10 09:22:52 [1783] - verbose: [hotswap] Router with id `plugins` replaced successfully
16/10 09:22:52 [1783] - verbose: [plugins] All plugins reloaded and rerouted
16/10 09:22:52 [1783] - verbose: [hotswap] Router with id `auth` replaced successfully
16/10 09:22:52 [1783] - info: Routes added

NodeBB reports as started however it won't listen to incoming connections. If I run ./nodebb stop the processes don't stop until I run killall node.

I've tried the following:

A git hard reset to v.1.6.x
./nodebb reset -p to reset the plugins

My production site is currently down. What's the recommended action to get it back up and running?

Sidenote: NodeBB runs at 100% CPU and won't stop processes without a reboot.

julian

That's odd. Did ./nodebb build complete without errors?

As 1.6.1 did not require any upgrade scripts, you can simply git reset --hard v1.6.0 to go back to v1.6.0 and be up and running again (assuming you were on v1.6.0 before).

In the future, we'd advise you to use a staging instance to test upgrades on before doing it on live instances.

nsuchy

I was able to successfully rollback to 1.6.0 however I'd still like to upgrade to 1.6.1 as soon as possible. Now that I'm all rolled back, what steps do you recommend I take to upgrade?

julian

I would make a copy of your database, and set up another installation of NodeBB for testing purposes... I've never seen an issue with NodeBB just refusing to start like that, so I don't know where to begin debugging.

nsuchy

@julian Staging server will be ready in 10 minutes. In SSH now I just need a few minutes to setup.

nsuchy

@julian Sorry about the delays, file transfers took a bit longer than expected. I have a staging server setup at https://staging.sinfulforums.net/login/2fa, if you'd like I can send you SSH Credentials to Staging. I'm fixing a few things regarding plugins. It still has certain plugins deactivated breaking login so I'm working on a fix for that then we can start working on an upgrade tactic.

julian

@nsuchy We could use that as part of the test... if you disable all plugins except the bundled ones, does it upgrade to v1.6.1 without issues

nsuchy

@julian Ok on staging I ran nodebb reset -p to deactivate all plugins, I ran th commands to upgrade, and now nodebb is running at 100% CPU on the staging server.

root@NodeBB-Staging:/var/www/html/NodeBB# curl -I http://127.0.0.1:4567
curl: (7) Failed to connect to 127.0.0.1 port 4567: Connection refused
root@NodeBB-Staging:/var/www/html/NodeBB# ./nodebb log

Hit Ctrl-C to exit

This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it under certain conditions.
For the full license, please visit: http://www.gnu.org/copyleft/gpl.html

Clustering enabled: Spinning up 1 process(es).

16/10 15:17:25 [3728] - info: Initializing NodeBB v1.6.1 https://staging.sinfulforums.net:443
16/10 15:17:25 [3728] - warn: You have no mongo username/password setup!
16/10 15:17:28 [3728] - info: [socket.io] Restricting access to origin: *:*
16/10 15:17:28 [3728] - info: Routes added

CPU and RAM at 100% utilization by the way.

julian

For all those wondering... @nsuchy had a really large (i.e. 2mb+ of pure text) blacklist.

...

... don't make your blacklist that large

Shard

@julian Ouch.

nsuchy

@shard I probably have the world's most unique edgecase

Sidenote for huge IP Ban Lists run the following bash script:

for i in `cat ./list.txt`; do echo 'deny' $i';' >> denyips.conf; done

list.txt being a list of IP addresses/ranges one IP / Range per line, denyips.conf being the file you give to the nginx proxy.

root@vps146875:/etc/nginx# wc -l ./denyips.conf
158740 ./denyips.conf

it took like a second to do it's thing very fast for so many lines.

Shard

Would that not better handled by your reverse proxy, software, or hardware firewall?

Sfner

I've got the same problem a while ago. Deactivating the broken plug-ins isn't enough: you have to uninstall (npm un <package-name>) the right one from your installation.

They usually show up in the logs from template building (node nodebb build) as errors or warns and in the dev console (node nodebb dev) as errors.

Good luck!