Forum down after restart

phenomlab

At this point, I'm totally stumped. The only other option I have available is to create a new instance and attach the current database to that to see if this resolves the issue or not.

Failing that, I'm at a total loss here.

phenomlab

@PitaJ @baris I've been looking into this more, and but when I start NodeBB I get this message

2021-09-26T19:52:22.547Z [4567/349055] - info: Enabling 'trust proxy'
2021-09-26T19:52:22.550Z [4567/349055] - error: NodeBB address in use, exiting...
Error: listen EADDRINUSE: address already in use 0.0.0.0:4567
    at Server.setupListenHandle [as _listen2] (net.js:1320:16)
    at listenInCluster (net.js:1368:12)
    at doListen (net.js:1505:7)
    at processTicksAndRejections (internal/process/task_queues.js:83:21)
2021-09-26T19:52:22.552Z [4567/349055] - error: uncaughtException: listen EADDRINUSE: address already in use 0.0.0.0:4567
Error: listen EADDRINUSE: address already in use 0.0.0.0:4567
3 restarts in 10 seconds, most likely an error on startup. Halting.

The problem is, the port isn't actively being used anywhere. I have another NodeBB instance running on 4568, but that's it.

Any thoughts ?

Thanks

PitaJ

@phenomlab I don't know what to tell you. It's not lying about the address being in use.

Does it work if you run node app? If not, it's definitely a rogue process left running on that port.

phenomlab

@pitaj thanks. Found the rogue process, and now I'm stuck with this.

sudonix@vps:~/nodebb$ ./nodebb start && ./nodebb log

Starting NodeBB
  "./nodebb stop" to stop the NodeBB server
  "./nodebb log" to view server output
  "./nodebb help" for more commands


Hit Ctrl-C to exit


    at processTicksAndRejections (internal/process/task_queues.js:83:21)
2021-09-26T20:43:20.339Z [4567/2900] - info: [app] Shutdown (SIGTERM/SIGINT) Initialised.
2021-09-26T20:43:20.340Z [4567/2900] - error: Error [ERR_SERVER_NOT_RUNNING]: Server is not running.
    at Server.close (net.js:1608:12)
    at Object.onceWrapper (events.js:519:28)
    at Server.emit (events.js:412:35)
    at emitCloseNT (net.js:1661:8)
    at processTicksAndRejections (internal/process/task_queues.js:81:21)
2021-09-26T20:43:20.346Z [4567/2893] - info: [reputation-rules] settings loaded
3 restarts in 10 seconds, most likely an error on startup. Halting.

So it looks like the process spawns and then immediately crashes. Trouble is I don't know where. It even does this with all plugins disabled.

phenomlab

@pitaj said in Forum down after restart:

Does it work if you run node app?

Yes

sudonix@vps:~/nodebb$ node app
2021-09-26T21:37:41.594Z [4567/5912] - info: NodeBB v1.18.3 Copyright (C) 2013-2021 NodeBB Inc.
2021-09-26T21:37:41.596Z [4567/5912] - info: This program comes with ABSOLUTELY NO WARRANTY.
2021-09-26T21:37:41.596Z [4567/5912] - info: This is free software, and you are welcome to redistribute it under certain conditions.
2021-09-26T21:37:41.596Z [4567/5912] - info: 
2021-09-26T21:37:41.597Z [4567/5912] - info: Initializing NodeBB v1.18.3 https://sudonix.com
2021-09-26T21:37:43.378Z [4567/5912] - info: [socket.io] Restricting access to origin: https://sudonix.com:*

2021-09-26T21:37:43.808Z [4567/5912] - warn: [plugins/load] The following plugins may not be compatible with your version of NodeBB. This may cause unintended behaviour or crashing. In the event of an unresponsive NodeBB caused by this plugin, run `./nodebb reset -p PLUGINNAME` to disable it.
  * nodebb-plugin-emoji

2021-09-26T21:37:43.931Z [4567/5912] - info: [plugins/spam-be-gone] Settings loaded
2021-09-26T21:37:43.943Z [4567/5912] - info: [reputation-rules] settings loaded
2021-09-26T21:37:43.971Z [4567/5912] - info: [api] Adding 1 route(s) to `api/v3/plugins`
2021-09-26T21:37:43.999Z [4567/5912] - info: [router] Routes added
2021-09-26T21:37:44.007Z [4567/5912] - info: NodeBB Ready
2021-09-26T21:37:44.009Z [4567/5912] - info: Enabling 'trust proxy'
2021-09-26T21:37:44.013Z [4567/5912] - info: NodeBB is now listening on: 0.0.0.0:4567

PitaJ

Does it still work with daemon:false or does that no longer help?

phenomlab

@pitaj that works fine

The only issue of course is that when performing a restart from the ACP I have to kill the process and restart it manually

PitaJ

@phenomlab it must be something to do with the daemon stuff.

https://github.com/NodeBB/NodeBB/blob/ac1b9692f2bccbf8420224f152535c238404ef53/loader.js#L217

Can you try adding console.logs through there to try and find where the script stops executing for you?

phenomlab

@pitaj I found this discussion from a while ago and it seems to be my exact issue. Sadly, there never was a fix provided

phenomlab

Interestingly, ./nodebb slog works (but doesn't survive a rebuild and restart) fine, but I still need to leave this running as it isn't a daemon

PitaJ

@phenomlab ./nodebb slog just runs ./nodebb start then shows the log with tail. Nothing fancy. When you quit the command, you're just exiting tail, NodeBB isn't stopped.

The "production" equivalent of ./nodebb dev is node app. There are many ways you can set it up to run as a daemon. For instance, setting up a systemd service.

gotwf

This post is deleted!

gotwf

@phenomlab Yo mon! Ye' be having just way too much fun!

@pitaj said in Forum down after restart:

There are many ways you can set it up to run as a daemon. For instance, setting up a systemd service.

Which is probably the way to go if you're using a systemd based distro

Running NodeBB - NodeBB Documentation

Documentation portal for NodeBB Forum Software

(docs.nodebb.org)

Rock on!

phenomlab

@pitaj @gotwf thanks guys. Much appreciated. I'll go down this route - probably using systemd to create a service.

phenomlab

@PitaJ interestingly, after starting NodeBB using ./NodeBB slog, if I then execute ./NodeBB stop I get

NodeBB is already stopped.

So think it's correct to say that as far as the daemon is concerned, it's not running even though it actually is.

PitaJ

@phenomlab glad that'll work for you.

Would you mind participating in some more debug for this down the line? There's an alternative package for starting the daemon that I'm wondering might work for you.

phenomlab

@pitaj Of course. Just let me know

Sampo2910

@phenomlab said in Forum down after restart:

@baris @PitaJ just tried an upgrade, and got this

Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.13.7-051307-lowlatency x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
Last login: Thu Sep 23 19:49:08 2021 from 77.101.200.87
sudonix@vps:~$ cd nodebb
sudonix@vps:~/nodebb$ git fetch
remote: Enumerating objects: 767, done.
remote: Counting objects: 100% (607/607), done.
remote: Compressing objects: 100% (91/91), done.
remote: Total 767 (delta 535), reused 580 (delta 516), pack-reused 160
Receiving objects: 100% (767/767), 196.89 KiB | 12.30 MiB/s, done.
Resolving deltas: 100% (576/576), completed with 254 local objects.
From https://github.com/NodeBB/NodeBB
 * [new branch]          dependabot/npm_and_yarn/install/commander-8.2.0         -> origin/dependabot/npm_and_yarn/install/commander-8.2.0
 + e0fda82b2...7502d5da8 dependabot/npm_and_yarn/install/connect-pg-simple-7.0.0 -> origin/dependabot/npm_and_yarn/install/connect-pg-simple-7.0.0  (forced update)
 * [new branch]          dependabot/npm_and_yarn/install/mongodb-4.1.2           -> origin/dependabot/npm_and_yarn/install/mongodb-4.1.2
 + b0787b593...a8ba14cdc dependabot/npm_and_yarn/install/postcss-clean-1.2.2     -> origin/dependabot/npm_and_yarn/install/postcss-clean-1.2.2  (forced update)
   0a56158bb..04ee5c0df  master                                                  -> origin/master
 + 4a5f10d2a...b638183dd renovate/mongodb-4.x                                    -> origin/renovate/mongodb-4.x  (forced update)
 * [new branch]          topic-linkbacks                                         -> origin/topic-linkbacks
   9410a73aa..1fad6f680  v1.17.x                                                 -> origin/v1.17.x
   854c078b7..9e5223697  v1.18.x                                                 -> origin/v1.18.x
 * [new tag]             v1.17.2-beta.8                                          -> v1.17.2-beta.8
 * [new tag]             v1.18.3                                                 -> v1.18.3
 * [new tag]             v1.17.2-beta.3                                          -> v1.17.2-beta.3
 * [new tag]             v1.17.2-beta.4                                          -> v1.17.2-beta.4
 * [new tag]             v1.17.2-beta.5                                          -> v1.17.2-beta.5
 * [new tag]             v1.17.2-beta.6                                          -> v1.17.2-beta.6
 * [new tag]             v1.17.2-beta.7                                          -> v1.17.2-beta.7
sudonix@vps:~/nodebb$ git reset --hard origin/v1.18.x
HEAD is now at 9e5223697 chore: incrementing version number - v1.18.3
sudonix@vps:~/nodebb$ ./nodebb upgrade

Updating NodeBB...

1. Updating package.json file with defaults...  OK

2. Bringing base dependencies up to date...  started
npm ERR! code ERESOLVE
npm ERR! ERESOLVE unable to resolve dependency tree
npm ERR! 
npm ERR! While resolving: [email protected]
npm ERR! Found: [email protected]
npm ERR! node_modules/mongodb
npm ERR!   mongodb@"3.7.1" from the root project
npm ERR! 
npm ERR! Could not resolve dependency:
npm ERR! peer mongodb@"^4.1.0" from [email protected]
npm ERR! node_modules/connect-mongo
npm ERR!   connect-mongo@"4.6.0" from the root project
npm ERR! 
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force, or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
npm ERR! 
npm ERR! See /home/sudonix/.npm/eresolve-report.txt for a full report.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/sudonix/.npm/_logs/2021-09-23T18_51_40_851Z-debug.log
Error installing dependencies!
message: Command failed: npm install --production
stdout: null
stderr: null
Error occurred during upgrade: Error: Command failed: npm install --production
    at checkExecSyncError (node:child_process:826:11)
    at Object.execSync (node:child_process:900:15)
    at Object.installAll (/home/sudonix/nodebb/src/cli/package-install.js:73:9)
    at Object.handler (/home/sudonix/nodebb/src/cli/upgrade.js:21:19)
    at runSteps (/home/sudonix/nodebb/src/cli/upgrade.js:54:16)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Object.runUpgrade [as upgrade] (/home/sudonix/nodebb/src/cli/upgrade.js:84:3)
2021-09-23T18:51:40.893Z [4567/78126] - error: uncaughtException: Command failed: npm install --production
Error: Command failed: npm install --production
    at checkExecSyncError (node:child_process:826:11)
    at Object.execSync (node:child_process:900:15)
    at Object.installAll (/home/sudonix/nodebb/src/cli/package-install.js:73:9)
    at Object.handler (/home/sudonix/nodebb/src/cli/upgrade.js:21:19)
    at runSteps (/home/sudonix/nodebb/src/cli/upgrade.js:54:16)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Object.runUpgrade [as upgrade] (/home/sudonix/nodebb/src/cli/upgrade.js:84:3) {"error":{"status":1,"signal":null,"output":[null,null,null],"pid":78133,"stdout":null,"stderr":null},"stack":"Error: Command failed: npm install --production\n    at checkExecSyncError (node:child_process:826:11)\n    at Object.execSync (node:child_process:900:15)\n    at Object.installAll (/home/sudonix/nodebb/src/cli/package-install.js:73:9)\n    at Object.handler (/home/sudonix/nodebb/src/cli/upgrade.js:21:19)\n    at runSteps (/home/sudonix/nodebb/src/cli/upgrade.js:54:16)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Object.runUpgrade [as upgrade] (/home/sudonix/nodebb/src/cli/upgrade.js:84:3)","exception":true,"date":"Thu Sep 23 2021 19:51:40 GMT+0100 (British Summer Time)","process":{"pid":78126,"uid":1021,"gid":1011,"cwd":"/home/sudonix/nodebb","execPath":"/usr/bin/node","version":"v16.10.0","argv":["/usr/bin/node","/home/sudonix/nodebb/nodebb","upgrade"],"memoryUsage":{"rss":67944448,"heapTotal":31772672,"heapUsed":21097072,"external":1831674,"arrayBuffers":926654}},"os":{"loadavg":[0.27,0.12,0.03],"uptime":85792.33},"tr:ace":[{"column":11,"file":"node:child_process","function":"checkExecSyncError","line":826,"method":null,"native":false},{"column":15,"file":"node:child_process","function":"Object.execSync","line":900,"method":"execSync","native":false},{"column":9,"file":"/home/sudonix/nodebb/src/cli/package-install.js","function":"Object.installAll","line":73,"method":"installAll","native":false},{"column":19,"file":"/home/sudonix/nodebb/src/cli/upgrade.js","function":"Object.handler","line":21,"method":"handler","native":false},{"column":16,"file":"/home/sudonix/nodebb/src/cli/upgrade.js","function":"runSteps","line":54,"method":null,"native":false},{"column":5,"file":"node:internal/process/task_queues","function":"processTicksAndRejections","line":96,"method":null,"native":false},{"column":3,"file":"/home/sudonix/nodebb/src/cli/upgrade.js","function":"async Object.runUpgrade [as upgrade]","line":84,"method":"runUpgrade [as upgrade]","native":false}]}
sudonix@vps:~/nodebb$

Any thoughts ?

Thanks

When i got that i error i used: npm install --save --legacy-peer-deps

That worked for me.

phenomlab

@sampo2910 thanks, although I think there is a post that mentions your have to downgrade to V6 which also worked for me.

phenomlab

@PitaJ Sorry I'm a little late responding here. I've used systemd as defined below, which works fine, and survives a rebuilt and restart

[Unit]
Description=Sudonix
Documentation=https://docs.nodebb.org
After=system.slice multi-user.target mongod.service

[Service]
Type=forking
User=sudonix

StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=nodebb

WorkingDirectory=/home/sudonix/nodebb
PIDFile=/home/sudonix/nodebb/pidfile
ExecStart=/usr/bin/env node loader.js
Restart=always

[Install]
WantedBy=multi-user.target

Many thanks for all the help and support. Greatly appreciated.