NodeBB won't restart



  • After upgrading to NodeBB 1.6.1 I am unable to start the service.

    ./nodebb dev output:

    root@vps146875:/var/www/sinfulforumsnet/NodeBB# ./nodebb dev
    
    NodeBB v1.6.1 Copyright (C) 2013-2014 NodeBB Inc.
    This program comes with ABSOLUTELY NO WARRANTY.
    This is free software, and you are welcome to redistribute it under certain conditions.
    For the full license, please visit: http://www.gnu.org/copyleft/gpl.html
    
    Clustering enabled: Spinning up 1 process(es).
    
    16/10 09:22:46 [1783] - verbose: * using configuration stored in: /var/www/sinfulforumsnet/NodeBB/config.json
    16/10 09:22:46 [1783] - info: Initializing NodeBB v1.6.1 https://sinfulforums.net:443
    16/10 09:22:46 [1783] - verbose: * using mongo store at 127.0.0.1:27017
    16/10 09:22:46 [1783] - verbose: * using themes stored in: /var/www/sinfulforumsnet/NodeBB/node_modules
    16/10 09:22:46 [1783] - warn: You have no mongo username/password setup!
    16/10 09:22:50 [1783] - verbose: [minifier] utilizing a maximum of 0 additional threads
    16/10 09:22:50 [1783] - verbose: Checking dependencies for outdated modules
    16/10 09:22:51 [1783] - verbose: [notifications.init] Registering jobs.
    16/10 09:22:51 [1783] - verbose: [user/jobs] (Re-)starting user jobs...
    16/10 09:22:51 [1783] - verbose: [user/jobs] Starting job (digest.daily)
    16/10 09:22:51 [1783] - verbose: [user/jobs] Starting job (digest.weekly)
    16/10 09:22:51 [1783] - verbose: [user/jobs] Starting job (digest.monthly)
    16/10 09:22:51 [1783] - verbose: [user/jobs] Starting job (reset.clean)
    16/10 09:22:51 [1783] - verbose: [user/jobs] 4 jobs started
    16/10 09:22:51 [1783] - verbose: [emailer] Setting up SMTP fallback transport
    16/10 09:22:51 [1783] - info: [themes] Using theme nodebb-theme-persona
    16/10 09:22:51 [1783] - verbose: [plugins] Initializing plugins system
    16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-composer-default
    16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-theme-persona
    16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-markdown
    16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-mentions
    16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-widget-essentials
    16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-rewards-essentials
    16/10 09:22:51 [1783] - verbose: [plugins] Found 3 sound file(s) for plugin nodebb-plugin-soundpack-default
    16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-soundpack-default
    16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-emoji-extended
    16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-emoji-one
    16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-2factor
    16/10 09:22:51 [1783] - warn: [plugins/nodebb-plugin-spam-be-gone] Hook `action:flag.create` is deprecated, please use `action:flags.create` instead.
    16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-spam-be-gone
    16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-emailer-smtp
    16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-dbsearch
    16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-emailer-sendgrid
    16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-imgur
    16/10 09:22:51 [1783] - verbose: [plugins] Loaded plugin: nodebb-plugin-desktop-notifications
    16/10 09:22:51 [1783] - info: [plugins] Plugins OK
    16/10 09:22:52 [1783] - info: [plugins/spam-be-gone] Settings loaded
    16/10 09:22:52 [1783] - verbose: [hotswap] Router with id `plugins` replaced successfully
    16/10 09:22:52 [1783] - verbose: [plugins] All plugins reloaded and rerouted
    16/10 09:22:52 [1783] - verbose: [hotswap] Router with id `auth` replaced successfully
    16/10 09:22:52 [1783] - info: Routes added
    

    NodeBB reports as started however it won't listen to incoming connections. If I run ./nodebb stop the processes don't stop until I run killall node.

    I've tried the following:

    • A git hard reset to v.1.6.x
    • ./nodebb reset -p to reset the plugins

    My production site is currently down. What's the recommended action to get it back up and running?

    Sidenote: NodeBB runs at 100% CPU and won't stop processes without a reboot.


  • Admin

    That's odd. Did ./nodebb build complete without errors?

    As 1.6.1 did not require any upgrade scripts, you can simply git reset --hard v1.6.0 to go back to v1.6.0 and be up and running again (assuming you were on v1.6.0 before).

    In the future, we'd advise you to use a staging instance to test upgrades on before doing it on live instances.



  • I was able to successfully rollback to 1.6.0 however I'd still like to upgrade to 1.6.1 as soon as possible. Now that I'm all rolled back, what steps do you recommend I take to upgrade?


  • Admin

    I would make a copy of your database, and set up another installation of NodeBB for testing purposes... I've never seen an issue with NodeBB just refusing to start like that, so I don't know where to begin debugging.



  • @julian Staging server will be ready in 10 minutes. In SSH now I just need a few minutes to setup.



  • @julian Sorry about the delays, file transfers took a bit longer than expected. I have a staging server setup at https://staging.sinfulforums.net/login/2fa, if you'd like I can send you SSH Credentials to Staging. I'm fixing a few things regarding plugins. It still has certain plugins deactivated breaking login so I'm working on a fix for that then we can start working on an upgrade tactic.


  • Admin

    @nsuchy We could use that as part of the test... if you disable all plugins except the bundled ones, does it upgrade to v1.6.1 without issues 🤔



  • @julian Ok on staging I ran nodebb reset -p to deactivate all plugins, I ran th commands to upgrade, and now nodebb is running at 100% CPU on the staging server.

    root@NodeBB-Staging:/var/www/html/NodeBB# curl -I http://127.0.0.1:4567
    curl: (7) Failed to connect to 127.0.0.1 port 4567: Connection refused
    root@NodeBB-Staging:/var/www/html/NodeBB# ./nodebb log
    
    Hit Ctrl-C to exit
    
    This program comes with ABSOLUTELY NO WARRANTY.
    This is free software, and you are welcome to redistribute it under certain conditions.
    For the full license, please visit: http://www.gnu.org/copyleft/gpl.html
    
    Clustering enabled: Spinning up 1 process(es).
    
    16/10 15:17:25 [3728] - info: Initializing NodeBB v1.6.1 https://staging.sinfulforums.net:443
    16/10 15:17:25 [3728] - warn: You have no mongo username/password setup!
    16/10 15:17:28 [3728] - info: [socket.io] Restricting access to origin: *:*
    16/10 15:17:28 [3728] - info: Routes added
    
    

    CPU and RAM at 100% utilization by the way.


  • Admin

    For all those wondering... @nsuchy had a really large (i.e. 2mb+ of pure text) blacklist.

    ...

    ... don't make your blacklist that large 😛



  • @julian Ouch.



  • @shard I probably have the world's most unique edgecase 🙂

    Sidenote for huge IP Ban Lists run the following bash script:

    for i in `cat ./list.txt`; do echo 'deny' $i';' >> denyips.conf; done
    

    list.txt being a list of IP addresses/ranges one IP / Range per line, denyips.conf being the file you give to the nginx proxy.

    root@vps146875:/etc/nginx# wc -l ./denyips.conf
    158740 ./denyips.conf
    

    it took like a second to do it's thing very fast for so many lines.



  • Would that not better handled by your reverse proxy, software, or hardware firewall?


  • GNU/Linux

    I've got the same problem a while ago. Deactivating the broken plug-ins isn't enough: you have to uninstall (npm un <package-name>) the right one from your installation.

    They usually show up in the logs from template building (node nodebb build) as errors or warns and in the dev console (node nodebb dev) as errors.

    Good luck!


Log in to reply
 


Star

Looks like your connection to NodeBB was lost, please wait while we try to reconnect.