Forum down after restart

phenomlab

@baris Thanks. Interestingly, it'll start with node app js ??

sudonix@vps:~/nodebb$ node app js
2021-09-22T21:26:50.523Z [4567/23090] - info: NodeBB v1.18.2 Copyright (C) 2013-2021 NodeBB Inc.
2021-09-22T21:26:50.524Z [4567/23090] - info: This program comes with ABSOLUTELY NO WARRANTY.
2021-09-22T21:26:50.524Z [4567/23090] - info: This is free software, and you are welcome to redistribute it under certain conditions.
2021-09-22T21:26:50.524Z [4567/23090] - info: 
2021-09-22T21:26:50.525Z [4567/23090] - info: Initializing NodeBB v1.18.2 https://sudonix.com
2021-09-22T21:26:52.683Z [4567/23090] - info: [socket.io] Restricting access to origin: https://sudonix.com:*

2021-09-22T21:26:53.138Z [4567/23090] - warn: [plugins/load] The following plugins may not be compatible with your version of NodeBB. This may cause unintended behaviour or crashing. In the event of an unresponsive NodeBB caused by this plugin, run `./nodebb reset -p PLUGINNAME` to disable it.
  * nodebb-plugin-emoji

2021-09-22T21:26:53.299Z [4567/23090] - info: [reputation-rules] settings loaded
2021-09-22T21:26:53.305Z [4567/23090] - info: [plugins/spam-be-gone] Settings loaded
2021-09-22T21:26:53.343Z [4567/23090] - info: [api] Adding 0 route(s) to `api/v3/plugins`
2021-09-22T21:26:53.370Z [4567/23090] - info: [router] Routes added
2021-09-22T21:26:53.377Z [4567/23090] - info: NodeBB Ready
2021-09-22T21:26:53.379Z [4567/23090] - info: Enabling 'trust proxy'
2021-09-22T21:26:53.382Z [4567/23090] - info: NodeBB is now listening on: 0.0.0.0:4567

phenomlab

@baris said in Forum down after restart:

I suggest disabling all custom plugins and themes and see if the issue is still there

I actually disabled everything with ./nodebb reset - exactly the same issue.

I'm happy for this to run in ./nodebb dev but will probably need to execute it as a cron job or something. I can't leave the remote SSH session running indefinitely for obvious reasons.

phenomlab

@baris said in Forum down after restart:

Also modify your config.json to start only a single process and see if that makes a difference.

Can you provide an explanation of what is needed to do this ?

<baris>

If your config.json has "port": [4567, 4568, 4569] then you change it to "port": "4567"

You say you disabled everything but your startup log still shows widgets.contentbetween I don't think that belongs in any of the core themes/plugins.

phenomlab

@baris said in Forum down after restart:

If your config.json has "port": [4567, 4568, 4569] then you change it to "port": "4567"

It doesn't. Just the one port

@baris said in Forum down after restart:

You say you disabled everything but your startup log still shows widgets.contentbetween I don't think that belongs in any of the core themes/plugins

I just enabled the widgets again. The content between notices are from modifications made in either widgets, or using customize. However, they've worked for months (actually since inception) without issues.

PitaJ

@phenomlab don't run in ./nodebb dev long term. It doesn't enforce websocket origin and therefore leaves your users susceptible to cross site websocket hijacking.

phenomlab

@pitaj Thanks. I read through the associated post you sent, and yes, it does sound like my particular issue but I am not seeing the need for a swap file on a system as large as mine.

The other NodeBB instance I have running is absolutely fine.

phenomlab

I'm curious to understand what exactly that error is I'm encountering and why NodeBB chooses to crash out at that point. Perhaps if I understood the reasoning, working towards a fix might be easier ?

Thanks for all the support thus far guys. Really appreciated.

phenomlab

Not sure I can even run in ./NodeBB dev short term as this will require an open connection to my server which isn't permitted (the open session will time out meaning NodeBB will stop)

phenomlab

What is the difference between ./NodeBB start and ./NodeBB dev ? Why does one work and not the other ?

PitaJ

@phenomlab I don't know what the error is, nor do I know why it's crashing there. The log looks incomplete but I don't know why that is either.

I'm as confused as you are. Searching for that be sendmail error results in nothing useful to me. I don't even know why sendmail would be involved in startup.

Maybe you're seeing outdated logs? Can you try renaming logs/output.log to something else and doing ./nodebb slog again?

Have you tried chowning the files to the user you're running as?

phenomlab

@pitaj very odd. I deleted the log file and it was regenerated with ./NodeBB start but it contains no entries and NodeBB still won't start

What permissions should be assigned to folders and files ? 755 and 664 ? The username is already correct as far as I can see.

Thanks

PitaJ

So it works of you run with node app, right? No errors when you run that?

Have you double checked to make sure all node processes are killed?

If you're running with a single port, you might try using screen or tmux to run with node app as a stopgap. That essentially the same as running with ./nodebb dev except all security checks stay in place.

You could also try setting "daemon": false in config.json and try ./nodebb start with that.

As for permissions, yeah make sure all files are owned by nodebb:nodebb and have 755 permissions.

phenomlab

@pitaj a bit of a positive development. I've added "daemon": false into config.json and have successfully started NodeBB. However, this then runs as a session which isn't great, but I can at least close the terminal window and the process continues to run without dying.

For now, this should be enough to keep the forum running until I figure out exactly why it won't start as a daemon. I noticed there's an upgrade to core so I'll try that also (1.18.3)

gotwf

@phenomlab Quite the saga here. Bummer for sure, especially on a production box. What prompted the restart? Perchance did that follow any system updates? If so, I presume you're hip enough to have already considered such but just a thought.

Royal pita but maybe time to cut your losses, export the db whilst you can, spin up a new vm and go from there? Tough call.

dunlix

What if you set up a new NodeBB from scratch but use the same database in your config and then move the uploads folder to the new one too?
Meaning all plugins and stuff like that will be nonexistant and it will be stock nodeBB installation. Then, you can reinstall your plugins one by one and your custom things too to see what exactly is causing this problem

Also i just read the latest post that you have (somewhat) fixed the issue but if your feeling up to a tedious job (lol) then try my solution

phenomlab

@baris @PitaJ just tried an upgrade, and got this

Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.13.7-051307-lowlatency x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
Last login: Thu Sep 23 19:49:08 2021 from 77.101.200.87
sudonix@vps:~$ cd nodebb
sudonix@vps:~/nodebb$ git fetch
remote: Enumerating objects: 767, done.
remote: Counting objects: 100% (607/607), done.
remote: Compressing objects: 100% (91/91), done.
remote: Total 767 (delta 535), reused 580 (delta 516), pack-reused 160
Receiving objects: 100% (767/767), 196.89 KiB | 12.30 MiB/s, done.
Resolving deltas: 100% (576/576), completed with 254 local objects.
From https://github.com/NodeBB/NodeBB
 * [new branch]          dependabot/npm_and_yarn/install/commander-8.2.0         -> origin/dependabot/npm_and_yarn/install/commander-8.2.0
 + e0fda82b2...7502d5da8 dependabot/npm_and_yarn/install/connect-pg-simple-7.0.0 -> origin/dependabot/npm_and_yarn/install/connect-pg-simple-7.0.0  (forced update)
 * [new branch]          dependabot/npm_and_yarn/install/mongodb-4.1.2           -> origin/dependabot/npm_and_yarn/install/mongodb-4.1.2
 + b0787b593...a8ba14cdc dependabot/npm_and_yarn/install/postcss-clean-1.2.2     -> origin/dependabot/npm_and_yarn/install/postcss-clean-1.2.2  (forced update)
   0a56158bb..04ee5c0df  master                                                  -> origin/master
 + 4a5f10d2a...b638183dd renovate/mongodb-4.x                                    -> origin/renovate/mongodb-4.x  (forced update)
 * [new branch]          topic-linkbacks                                         -> origin/topic-linkbacks
   9410a73aa..1fad6f680  v1.17.x                                                 -> origin/v1.17.x
   854c078b7..9e5223697  v1.18.x                                                 -> origin/v1.18.x
 * [new tag]             v1.17.2-beta.8                                          -> v1.17.2-beta.8
 * [new tag]             v1.18.3                                                 -> v1.18.3
 * [new tag]             v1.17.2-beta.3                                          -> v1.17.2-beta.3
 * [new tag]             v1.17.2-beta.4                                          -> v1.17.2-beta.4
 * [new tag]             v1.17.2-beta.5                                          -> v1.17.2-beta.5
 * [new tag]             v1.17.2-beta.6                                          -> v1.17.2-beta.6
 * [new tag]             v1.17.2-beta.7                                          -> v1.17.2-beta.7
sudonix@vps:~/nodebb$ git reset --hard origin/v1.18.x
HEAD is now at 9e5223697 chore: incrementing version number - v1.18.3
sudonix@vps:~/nodebb$ ./nodebb upgrade

Updating NodeBB...

1. Updating package.json file with defaults...  OK

2. Bringing base dependencies up to date...  started
npm ERR! code ERESOLVE
npm ERR! ERESOLVE unable to resolve dependency tree
npm ERR! 
npm ERR! While resolving: [email protected]
npm ERR! Found: [email protected]
npm ERR! node_modules/mongodb
npm ERR!   mongodb@"3.7.1" from the root project
npm ERR! 
npm ERR! Could not resolve dependency:
npm ERR! peer mongodb@"^4.1.0" from [email protected]
npm ERR! node_modules/connect-mongo
npm ERR!   connect-mongo@"4.6.0" from the root project
npm ERR! 
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force, or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
npm ERR! 
npm ERR! See /home/sudonix/.npm/eresolve-report.txt for a full report.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/sudonix/.npm/_logs/2021-09-23T18_51_40_851Z-debug.log
Error installing dependencies!
message: Command failed: npm install --production
stdout: null
stderr: null
Error occurred during upgrade: Error: Command failed: npm install --production
    at checkExecSyncError (node:child_process:826:11)
    at Object.execSync (node:child_process:900:15)
    at Object.installAll (/home/sudonix/nodebb/src/cli/package-install.js:73:9)
    at Object.handler (/home/sudonix/nodebb/src/cli/upgrade.js:21:19)
    at runSteps (/home/sudonix/nodebb/src/cli/upgrade.js:54:16)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Object.runUpgrade [as upgrade] (/home/sudonix/nodebb/src/cli/upgrade.js:84:3)
2021-09-23T18:51:40.893Z [4567/78126] - error: uncaughtException: Command failed: npm install --production
Error: Command failed: npm install --production
    at checkExecSyncError (node:child_process:826:11)
    at Object.execSync (node:child_process:900:15)
    at Object.installAll (/home/sudonix/nodebb/src/cli/package-install.js:73:9)
    at Object.handler (/home/sudonix/nodebb/src/cli/upgrade.js:21:19)
    at runSteps (/home/sudonix/nodebb/src/cli/upgrade.js:54:16)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Object.runUpgrade [as upgrade] (/home/sudonix/nodebb/src/cli/upgrade.js:84:3) {"error":{"status":1,"signal":null,"output":[null,null,null],"pid":78133,"stdout":null,"stderr":null},"stack":"Error: Command failed: npm install --production\n    at checkExecSyncError (node:child_process:826:11)\n    at Object.execSync (node:child_process:900:15)\n    at Object.installAll (/home/sudonix/nodebb/src/cli/package-install.js:73:9)\n    at Object.handler (/home/sudonix/nodebb/src/cli/upgrade.js:21:19)\n    at runSteps (/home/sudonix/nodebb/src/cli/upgrade.js:54:16)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Object.runUpgrade [as upgrade] (/home/sudonix/nodebb/src/cli/upgrade.js:84:3)","exception":true,"date":"Thu Sep 23 2021 19:51:40 GMT+0100 (British Summer Time)","process":{"pid":78126,"uid":1021,"gid":1011,"cwd":"/home/sudonix/nodebb","execPath":"/usr/bin/node","version":"v16.10.0","argv":["/usr/bin/node","/home/sudonix/nodebb/nodebb","upgrade"],"memoryUsage":{"rss":67944448,"heapTotal":31772672,"heapUsed":21097072,"external":1831674,"arrayBuffers":926654}},"os":{"loadavg":[0.27,0.12,0.03],"uptime":85792.33},"tr:ace":[{"column":11,"file":"node:child_process","function":"checkExecSyncError","line":826,"method":null,"native":false},{"column":15,"file":"node:child_process","function":"Object.execSync","line":900,"method":"execSync","native":false},{"column":9,"file":"/home/sudonix/nodebb/src/cli/package-install.js","function":"Object.installAll","line":73,"method":"installAll","native":false},{"column":19,"file":"/home/sudonix/nodebb/src/cli/upgrade.js","function":"Object.handler","line":21,"method":"handler","native":false},{"column":16,"file":"/home/sudonix/nodebb/src/cli/upgrade.js","function":"runSteps","line":54,"method":null,"native":false},{"column":5,"file":"node:internal/process/task_queues","function":"processTicksAndRejections","line":96,"method":null,"native":false},{"column":3,"file":"/home/sudonix/nodebb/src/cli/upgrade.js","function":"async Object.runUpgrade [as upgrade]","line":84,"method":"runUpgrade [as upgrade]","native":false}]}
sudonix@vps:~/nodebb$

Any thoughts ?

Thanks

<baris>

@phenomlab Sounds like this https://community.nodebb.org/topic/16029/yet-another-error-upon-upgrade/2?_=1632423543060.

phenomlab

@baris thanks

phenomlab

@baris @pitaj after upgrading to 1.18.3 I now get this when attempting to start NodeBB (as daemon)

    at processTicksAndRejections (node:internal/process/task_queues:84:21)
2021-09-23T19:01:07.217Z [4567/78763] - info: [app] Shutdown (SIGTERM/SIGINT) Initialised.
2021-09-23T19:01:07.218Z [4567/78763] - error: Error [ERR_SERVER_NOT_RUNNING]: Server is not running.
    at new NodeError (node:internal/errors:371:5)
    at Server.close (node:net:1613:12)
    at Object.onceWrapper (node:events:509:28)
    at Server.emit (node:events:402:35)
    at emitCloseNT (node:net:1666:8)
    at processTicksAndRejections (node:internal/process/task_queues:82:21)
3 restarts in 10 seconds, most likely an error on startup. Halting.