Forum is not available anymore

[Nicolas]

Hello guys,

Here is a slightly weird situation. NodeBB could stop working after a long run, and it is not a problem. For some reason I have discovered today NodeBB in downstate, it could happen from time to time after several months of the run, but I could not revive it today.

As always, go to the NodeBB directory, run ./nodebb start I see the process is running, but the forum is not accessible anymore. I did not change anything for a very long time.

Surprisingly, when I do ./nodebb dev I can access the forum without any issues, but ./nodebb start does not provide expected results.

Any ideas?

Thanks,
Nicolas

[PitaJ]

@nicolas have you tried killing all node processes, then starting?

[Nicolas]

Yes. To make sure I have restarted the server even.

Why ./nodebb dev works? There is something less trivial in place...

The most confusing part for me, I did not touch the server for the past 2-3 months.

[baris]

Can you post some more info? config.json nodebb/nodejs version etc.

Also does it work when you run node app.js ?

[Nicolas]

node app.js does not work, the same results as ./nodebb start

I did not change the configuration for NodeBB and Nginx, it's the same from 2017

config.json the most important part

"url": "https://forum.xyz.in.ua",
"port": "8081",

NodeBB: 1.0.2
NodeJS: 5.9.1

Also did not change them for a long period of time. Literally no changes in the system.

[gotwf]

@nicolas None at all, even beyond scope of nodebb?

[Nicolas]

@gotwf none, every 3 months login and update certificates through letsencrypt. That is all.

[PitaJ]

What happens when you attempt to access with ./nodebb start? Do you get an error? Can you try sending a request to your forum with curl on the server (port 8081)?

Do you see anything in the logs? With ./nodebb slog do you see it start up normally?

[Nicolas]

@pitaj when I attempt to connect with ./nodebb start it rejects the connection

2021/03/03 03:25:54 [error] 4264#0: *54 connect() failed (111: Connection refused) while connecting to upstream, client: 98.212.156.155, server: forum.xyz.in.ua, request: "GET /socket.io/?EIO=3&transport=polling&t=NVs7j_e HTTP/1.1", upstream: "http://127.0.0.1:8081/socket.io/?EIO=3&transport=polling&t=NVs7j_e", host: "forum.xyz.in.ua", referrer: "https://forum.xyz.in.ua/"

NodeBB logs are OK:

3/3 03:26 [5099] - info: NodeBB Ready
3/3 03:26 [5099] - info: Enabling 'trust proxy'
3/3 03:26 [5099] - info: NodeBB is now listening on: 0.0.0.0:8081

I did not touch Nginx config. It's the same without any changes from 2017.

[omega]

Hmmm... has it been hacked?

Any strange user regs before it went down?

Maybe you can review the last few users who roistered. If it is open. Just a thought. I've had NodeBB go down like that. No time to check it out. Nothing works like it should. Wasn't an important install.

How it woudl happen is, admin woudl approve a user. Since this would be an no users, this user is the first. Within about 2/3 mins of approving use.. BANG!

NodeBb goes down. Nothing is working like it should, in a similar way as to you are describing.

[PitaJ]

@nicolas

Can you try sending a request directly to NodeBB with curl on your server? (localhost:8081)

Where was that error reported? Nginx logs?

Posting the last few lines of your startup log isn't really helpful, can you paste the whole thing? Post a startup log from ./nodebb dev while you're at it for comparison.

Can you try attaching NodeBB to a different port?

Have you tried deleting node_modules and then npm install? On your version you might need to reinstall your plugins again.

[Nicolas]

@pitaj curl result is similar to Nginx

$ curl localhost:8081
Cannot GET /

Connection refused error that I have posted before, from Nginx Error logs.

Considering the situation, currently, I'm not messing with dependencies or configs that would work without any issues. I did not touch NPM Modules, did not try to remap port.

By the way, curl localhost:8081 for NodeBB that I run in Dev mode ./nodebb dev returns HTML, i.e. forum functions

What so special happens with the production start of the NodeBB that could stop working in such a fashion?

@omega interesting theory, I will see if any suspicious users were created last couple of days. But I'm the only admin.

@PitaJ thank you for brainstorming the potential issues. Here are full logs from NodeBB Start:

NodeBB v1.0.2 Copyright (C) 2013-2014 NodeBB Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it under certain conditions.
For the full license, please visit: http://www.gnu.org/copyleft/gpl.html

Clustering enabled: Spinning up 1 process(es).

3/3 23:36 [15691] - info: Time: Wed Mar 03 2021 23:36:57 GMT+0000 (UTC)
3/3 23:36 [15691] - info: Initializing NodeBB v1.0.2
3/3 23:36 [15691] - info: [database] Checking database indices.
3/3 23:36 [15691] - warn: [socket.io] Clustering detected, you are advised to configure Redis as a websocket store.

3/3 23:36 [15691] - warn: [plugins/load] The following plugins may not be compatible with your version of NodeBB. This may cause unintended behaviour or crashing. In the event of an unresponsive NodeBB caused by this plugin, run `./nodebb reset -p PLUGINNAME` to disable it.
  * nodebb-plugin-ns-points
  * nodebb-plugin-emailer-mailgun

3/3 23:37 [15691] - info: NodeBB Ready
3/3 23:37 [15691] - info: Enabling 'trust proxy'
3/3 23:37 [15691] - info: NodeBB is now listening on: 0.0.0.0:8081

P.S.
Warning for Dev mode is the same, and it works...

[Nicolas]

@omega Unlikely it was hacked, it's low profile forum.

[Nicolas]

Cannot GET /

Is it Express error output? Why would Express suddenly start to be unhappy about the environment?..

[Nicolas]

@baris any ideas what would be the most useful place in the main app to put the console.log to troubleshoot Express, why it does not blurp output?

[PitaJ]

@nicolas that is an express error, which is why I recommended reinstalling dependencies. I'm guessing this was just caused by a random file corruption that's only visible in production mode.

[Nicolas]

@pitaj is it possible? Does Express mutate its source files?

Touching Node Modules with such an old Node version without lock file can become very deadly for the forum and current setup.

[baris]

The files in node_modules could change if someone installs a plugin in the ACP. Does this affect all routes? Does loading something like forum.com/api/config work?

What active plugins do you have?

[PitaJ]

@nicolas huh? Mutating source files? I'm talking about cosmic rays flipping bits here.

[Nicolas]

@baris /api/config - does not work.

$ curl localhost:8081/api/config
Cannot GET /api/config

Plugins. logs from Dev mode:

4/3 00:42 [18479] - info: [themes] Using theme nodebb-theme-overseers
4/3 00:42 [18479] - verbose: [plugins] Initializing plugins system
4/3 00:42 [18479] - verbose: [plugins] Found 5 js file(s) for plugin nodebb-theme-overseers
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-theme-overseers
4/3 00:42 [18479] - verbose: [plugins] Found 1 css file(s) for plugin nodebb-widget-essentials
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-widget-essentials
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-soundpack-default
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-mentions
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-mentions
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-mentions
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-markdown
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-markdown
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-markdown
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-markdown
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-emoji-apple
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-emoji-extended
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-emoji-extended
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-emoji-extended
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-composer-default
4/3 00:42 [18479] - verbose: [plugins] Found 11 js file(s) for plugin nodebb-plugin-composer-default
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-composer-default
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-composer-default
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-api
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-ns-likes
4/3 00:42 [18479] - verbose: [plugins] Found 2 js file(s) for plugin nodebb-plugin-ns-likes
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-likes
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-login
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-ns-points
4/3 00:42 [18479] - verbose: [plugins] Found 2 js file(s) for plugin nodebb-plugin-ns-points
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-points
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-slugify
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-write-api
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-write-api
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-write-api
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-ns-embed
4/3 00:42 [18479] - verbose: [plugins] Found 0 js file(s) for plugin nodebb-plugin-ns-embed
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-ns-embed
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-embed
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-dbsearch
4/3 00:42 [18479] - verbose: [plugins] Found 3 less file(s) for plugin nodebb-plugin-ns-custom-fields
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-ns-custom-fields
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-ns-custom-fields
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-custom-fields
4/3 00:42 [18479] - verbose: [plugins] Found 3 less file(s) for plugin nodebb-plugin-ns-awards
4/3 00:42 [18479] - verbose: [plugins] Found 0 js file(s) for plugin nodebb-plugin-ns-awards
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-ns-awards
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-awards
4/3 00:42 [18479] - verbose: [plugins] Found 0 css file(s) for plugin nodebb-widget-ns-birthdays
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-widget-ns-birthdays
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-widget-ns-birthdays
4/3 00:42 [18479] - verbose: [plugins] Found 0 css file(s) for plugin nodebb-widget-ns-stats
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-widget-ns-stats
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-widget-ns-stats
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-emailer-mailgun
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-google-analytics
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-google-analytics

4/3 00:42 [18479] - warn: [plugins/load] The following plugins may not be compatible with your version of NodeBB. This may cause unintended behaviour or crashing. In the event of an unresponsive NodeBB caused by this plugin, run `./nodebb reset -p PLUGINNAME` to disable it.
  * nodebb-plugin-ns-points
  * nodebb-plugin-emailer-mailgun

4/3 00:42 [18479] - info: [plugins] Plugins OK