Forum is not available anymore

[gotwf]

@nicolas None at all, even beyond scope of nodebb?

[Nicolas]

@gotwf none, every 3 months login and update certificates through letsencrypt. That is all.

[PitaJ]

What happens when you attempt to access with ./nodebb start? Do you get an error? Can you try sending a request to your forum with curl on the server (port 8081)?

Do you see anything in the logs? With ./nodebb slog do you see it start up normally?

[Nicolas]

@pitaj when I attempt to connect with ./nodebb start it rejects the connection

2021/03/03 03:25:54 [error] 4264#0: *54 connect() failed (111: Connection refused) while connecting to upstream, client: 98.212.156.155, server: forum.xyz.in.ua, request: "GET /socket.io/?EIO=3&transport=polling&t=NVs7j_e HTTP/1.1", upstream: "http://127.0.0.1:8081/socket.io/?EIO=3&transport=polling&t=NVs7j_e", host: "forum.xyz.in.ua", referrer: "https://forum.xyz.in.ua/"

NodeBB logs are OK:

3/3 03:26 [5099] - info: NodeBB Ready
3/3 03:26 [5099] - info: Enabling 'trust proxy'
3/3 03:26 [5099] - info: NodeBB is now listening on: 0.0.0.0:8081

I did not touch Nginx config. It's the same without any changes from 2017.

[omega]

Hmmm... has it been hacked?

Any strange user regs before it went down?

Maybe you can review the last few users who roistered. If it is open. Just a thought. I've had NodeBB go down like that. No time to check it out. Nothing works like it should. Wasn't an important install.

How it woudl happen is, admin woudl approve a user. Since this would be an no users, this user is the first. Within about 2/3 mins of approving use.. BANG!

NodeBb goes down. Nothing is working like it should, in a similar way as to you are describing.

[PitaJ]

@nicolas

Can you try sending a request directly to NodeBB with curl on your server? (localhost:8081)

Where was that error reported? Nginx logs?

Posting the last few lines of your startup log isn't really helpful, can you paste the whole thing? Post a startup log from ./nodebb dev while you're at it for comparison.

Can you try attaching NodeBB to a different port?

Have you tried deleting node_modules and then npm install? On your version you might need to reinstall your plugins again.

[Nicolas]

@pitaj curl result is similar to Nginx

$ curl localhost:8081
Cannot GET /

Connection refused error that I have posted before, from Nginx Error logs.

Considering the situation, currently, I'm not messing with dependencies or configs that would work without any issues. I did not touch NPM Modules, did not try to remap port.

By the way, curl localhost:8081 for NodeBB that I run in Dev mode ./nodebb dev returns HTML, i.e. forum functions

What so special happens with the production start of the NodeBB that could stop working in such a fashion?

@omega interesting theory, I will see if any suspicious users were created last couple of days. But I'm the only admin.

@PitaJ thank you for brainstorming the potential issues. Here are full logs from NodeBB Start:

NodeBB v1.0.2 Copyright (C) 2013-2014 NodeBB Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it under certain conditions.
For the full license, please visit: http://www.gnu.org/copyleft/gpl.html

Clustering enabled: Spinning up 1 process(es).

3/3 23:36 [15691] - info: Time: Wed Mar 03 2021 23:36:57 GMT+0000 (UTC)
3/3 23:36 [15691] - info: Initializing NodeBB v1.0.2
3/3 23:36 [15691] - info: [database] Checking database indices.
3/3 23:36 [15691] - warn: [socket.io] Clustering detected, you are advised to configure Redis as a websocket store.

3/3 23:36 [15691] - warn: [plugins/load] The following plugins may not be compatible with your version of NodeBB. This may cause unintended behaviour or crashing. In the event of an unresponsive NodeBB caused by this plugin, run `./nodebb reset -p PLUGINNAME` to disable it.
  * nodebb-plugin-ns-points
  * nodebb-plugin-emailer-mailgun

3/3 23:37 [15691] - info: NodeBB Ready
3/3 23:37 [15691] - info: Enabling 'trust proxy'
3/3 23:37 [15691] - info: NodeBB is now listening on: 0.0.0.0:8081

P.S.
Warning for Dev mode is the same, and it works...

[Nicolas]

@omega Unlikely it was hacked, it's low profile forum.

[Nicolas]

Cannot GET /

Is it Express error output? Why would Express suddenly start to be unhappy about the environment?..

[Nicolas]

@baris any ideas what would be the most useful place in the main app to put the console.log to troubleshoot Express, why it does not blurp output?

[PitaJ]

@nicolas that is an express error, which is why I recommended reinstalling dependencies. I'm guessing this was just caused by a random file corruption that's only visible in production mode.

[Nicolas]

@pitaj is it possible? Does Express mutate its source files?

Touching Node Modules with such an old Node version without lock file can become very deadly for the forum and current setup.

[baris]

The files in node_modules could change if someone installs a plugin in the ACP. Does this affect all routes? Does loading something like forum.com/api/config work?

What active plugins do you have?

[PitaJ]

@nicolas huh? Mutating source files? I'm talking about cosmic rays flipping bits here.

[Nicolas]

@baris /api/config - does not work.

$ curl localhost:8081/api/config
Cannot GET /api/config

Plugins. logs from Dev mode:

4/3 00:42 [18479] - info: [themes] Using theme nodebb-theme-overseers
4/3 00:42 [18479] - verbose: [plugins] Initializing plugins system
4/3 00:42 [18479] - verbose: [plugins] Found 5 js file(s) for plugin nodebb-theme-overseers
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-theme-overseers
4/3 00:42 [18479] - verbose: [plugins] Found 1 css file(s) for plugin nodebb-widget-essentials
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-widget-essentials
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-soundpack-default
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-mentions
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-mentions
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-mentions
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-markdown
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-markdown
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-markdown
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-markdown
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-emoji-apple
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-emoji-extended
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-emoji-extended
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-emoji-extended
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-composer-default
4/3 00:42 [18479] - verbose: [plugins] Found 11 js file(s) for plugin nodebb-plugin-composer-default
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-composer-default
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-composer-default
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-api
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-ns-likes
4/3 00:42 [18479] - verbose: [plugins] Found 2 js file(s) for plugin nodebb-plugin-ns-likes
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-likes
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-login
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-ns-points
4/3 00:42 [18479] - verbose: [plugins] Found 2 js file(s) for plugin nodebb-plugin-ns-points
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-points
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-slugify
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-write-api
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-write-api
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-write-api
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-ns-embed
4/3 00:42 [18479] - verbose: [plugins] Found 0 js file(s) for plugin nodebb-plugin-ns-embed
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-ns-embed
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-embed
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-dbsearch
4/3 00:42 [18479] - verbose: [plugins] Found 3 less file(s) for plugin nodebb-plugin-ns-custom-fields
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-ns-custom-fields
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-ns-custom-fields
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-custom-fields
4/3 00:42 [18479] - verbose: [plugins] Found 3 less file(s) for plugin nodebb-plugin-ns-awards
4/3 00:42 [18479] - verbose: [plugins] Found 0 js file(s) for plugin nodebb-plugin-ns-awards
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-ns-awards
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-awards
4/3 00:42 [18479] - verbose: [plugins] Found 0 css file(s) for plugin nodebb-widget-ns-birthdays
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-widget-ns-birthdays
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-widget-ns-birthdays
4/3 00:42 [18479] - verbose: [plugins] Found 0 css file(s) for plugin nodebb-widget-ns-stats
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-widget-ns-stats
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-widget-ns-stats
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-emailer-mailgun
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-google-analytics
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-google-analytics

4/3 00:42 [18479] - warn: [plugins/load] The following plugins may not be compatible with your version of NodeBB. This may cause unintended behaviour or crashing. In the event of an unresponsive NodeBB caused by this plugin, run `./nodebb reset -p PLUGINNAME` to disable it.
  * nodebb-plugin-ns-points
  * nodebb-plugin-emailer-mailgun

4/3 00:42 [18479] - info: [plugins] Plugins OK

[Nicolas]

/api/config in Dev Mode - works

[psychobunny]

@nicolas though I don't want to immediately suggest you upgrade from NodeBB v1.0.2 (that's a LONG time ago! haha... welcome back btw!)

I wonder if you ran npm i etc and that upgraded you to packages that are no longer compatible with that version of core

[Nicolas]

I did not do npm install, I deactivated at this point 2 plugins that could depend on external services like Captcha and Twitch Monitor.

Here is an example of Express, if I would install node modules likely express would update also:

[Nicolas]

Any ideas why ./nodebb start does not work, and ./nodebb dev works?
Anything I could do to rebuild something in a less harmful way to try ./nodebb start?
Any suggestions why Express does not work in production mode?

[Nicolas]

So I'm debugging Express, in production, Route registration does not happen. For example, in debug mode I see logs like:

  express:router:route get /user/:userslug/edit/password +0ms
  express:router:layer new / +0ms
  express:router:route get /user/:userslug/edit/password +0ms
  express:router:layer new / +0ms
  express:router:route get /user/:userslug/edit/password +0ms
  express:router:layer new / +0ms
  express:router:route new /api/user/:userslug/edit/password +0ms
  express:router:layer new /api/user/:userslug/edit/password +0ms
  express:router:route get /api/user/:userslug/edit/password +0ms
  express:router:layer new / +0ms
  express:router:route get /api/user/:userslug/edit/password +0ms
  express:router:layer new / +0ms
  express:router:route get /api/user/:userslug/edit/password +0ms
  express:router:layer new / +0ms
  express:router:route get /api/user/:userslug/edit/password +0ms
  express:router:layer new / +0ms
  express:router:route get /api/user/:userslug/edit/password +0ms
  express:router:layer new / +0ms
  express:router:route new /user/:userslug/settings +0ms
  express:router:layer new /user/:userslug/settings +0ms
  express:router:route get /user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route get /user/:userslug/settings +1ms
  express:router:layer new / +0ms
  express:router:route get /user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route get /user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route get /user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route get /user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route get /user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route new /api/user/:userslug/settings +0ms
  express:router:layer new /api/user/:userslug/settings +0ms
  express:router:route get /api/user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route get /api/user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route get /api/user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route get /api/user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route get /api/user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route new /user/:userslug/session/:uuid +0ms
  express:router:layer new /user/:userslug/session/:uuid +0ms
  express:router:route delete /user/:userslug/session/:uuid +0ms

But in production mode I can see only initial Express operations:

express:application set "x-powered-by" to true +0ms
  express:application set "etag" to 'weak' +1ms
  express:application set "etag fn" to [Function: wetag] +1ms
  express:application set "env" to 'development' +0ms
  express:application set "query parser" to 'extended' +0ms
  express:application set "query parser fn" to [Function: parseExtendedQueryString] +0ms
  express:application set "subdomain offset" to 2 +0ms
  express:application set "trust proxy" to false +0ms
  express:application set "trust proxy fn" to [Function: trustNone] +1ms
  express:application booting in development mode +0ms
  express:application set "view" to [Function: View] +0ms
  express:application set "views" to '/web/overseers/node/forum/views' +0ms
  express:application set "jsonp callback name" to 'callback' +0ms
  express:application set "view engine" to 'tpl' +85ms
  express:application set "views" to '/web/overseers/node/forum/public/templates' +1ms
  express:application set "json spaces" to 0 +0ms
  express:router use / query +0ms
  express:router:layer new / +1ms
  express:router use / expressInit +0ms
  express:router:layer new / +0ms
  express:router use / <anonymous> +0ms
  express:router:layer new / +0ms
  express:application set "view cache" to true +1ms
  express:router use / compression +0ms
  express:router:layer new / +0ms
  express:router use  favicon +0ms
  express:router:layer new  +0ms
  express:router use /apple-touch-icon <anonymous> +0ms
  express:router:layer new /apple-touch-icon +0ms
  express:router use / urlencodedParser +26ms
  express:router:layer new / +1ms
  express:router use / jsonParser +0ms
  express:router:layer new / +0ms
  express:router use / cookieParser +0ms
  express:router:layer new / +0ms
  express:router use / <anonymous> +1ms
  express:router:layer new / +0ms
  express:router use / session +0ms
  express:router:layer new / +0ms
  express:router use / <anonymous> +0ms
  express:router:layer new / +0ms
  express:router use / <anonymous> +1ms
  express:router:layer new / +0ms
  express:router use / initialize +0ms
  express:router:layer new / +0ms
  express:router use / authenticate +0ms
  express:router:layer new / +0ms
  express:router use / <anonymous> +0ms
  express:router:layer new / +0ms
  express:router use / <anonymous> +1ms
  express:router:layer new / +0ms
  express:application set "cache" to true +1ms
  express:application set "minification" to true +0ms

That is why even the basic root route does not work since none of them are registered. Any hints where to look next?