node install borked itself or I think it did... maybe nginx, any hunting tips?

omega

A nodebb install went down on me - everything looks fine, server is running, nginx is running, nodes is running, but the URL is not resolving - cannot find server - which first thought is nginx... systemctl status nginx and it's up and running... - I did check the log and there was. a message about ns-embed plugin, I disabled that and rebuilt nodebb via cmd line but still no show, even though it's boots up and runs running according to terminal output.

I'll have a look at dev, but on quick look the vitals seem fine but now show... first for me. Maybe redis is the problem, but no errors on node startup, I haven't checked that, any tips there much appreciated.

Still think it's nginx... hmmm

nodebb v1.12.x

omega

I fixed the PID error in nginx, but that wasn't it.

Here is a section from the dev output, I think it's our old friend, EADDRINUSE error:

[4567/1345] - error: NodeBB address in use, exiting...listen **EADDRINUSE: address already in use 0.0.0.0:4567** {"code":"EADDRINUSE","errno":"EADDRINUSE","syscall":"listen","address":"0.0.0.0","port":4567,"stack":"Error: listen EADDRINUSE: address already in use 0.0.0.0:4567\n    at Server.setupListenHandle [as _listen2] (net.js:1259:14)\n    at listenInCluster (net.js:1307:12)\n    at doListen (net.js:1446:7)\n    at processTicksAndRejections (internal/process/task_queues.js:81:17)"}
>2020-01-15T23:08:48.753Z [4567/1345] - error: uncaughtException: listen EADDRINUSE: address already in use 0.0.0.0:4567
>Error: listen EADDRINUSE: address already in use 0.0.0.0:4567
>    at Server.setupListenHandle [as _listen2] (net.js:1259:14)
>    at listenInCluster (net.js:1307:12)
>    at doListen (net.js:1446:7)
>    at processTicksAndRejections (internal/process/task_queues.js:81:17) {"error":{"code":"EADDRINUSE","errno":"EADDRINUSE","syscall":"listen","address":"0.0.0.0","port":4567},"stack":"Error: listen EADDRINUSE: address already in use 0.0.0.0:4567\n    at Server.setupListenHandle [as _listen2] (net.js:1259:14)\n    at listenInCluster (net.js:1307:12)\n    at doListen (net.js:1446:7)\n    at processTicksAndRejections (internal/process/task_queues.js:81:17)","exception":true,"date":"Wed Jan 15 2020 23:08:48 GMT+0000 (Coordinated Universal Time)","process":{"pid":1345,"uid":1000,"gid":1000,"cwd":"/home/node/nodebb","execPath":"/usr/bin/node","version":"v11.15.0","argv":["/usr/bin/node","/home/node/nodebb/app.js"],"memoryUsage":{"rss":148332544,"heapTotal":110985216,"heapUsed":85905376,"external":692996}},"os":{"loadavg":[0.666015625,0.30126953125,0.10986328125],"uptime":410},"trace":[{"column":14,"file":"net.js","function":"Server.setupListenHandle [as _listen2]","line":1259,"method":"setupListenHandle [as _listen2]","native":false},{"column":12,"file":"net.js","function":"listenInCluster","line":1307,"method":null,"native":false},{"column":7,"file":"net.js","function":"doListen","line":1446,"method":null,"native":false},{"column":17,"file":"internal/process/task_queues.js","function":"processTicksAndRejections","line":81,"method":null,"native":false}]}
>2020-01-15T23:08:48.754Z [4567/1345] - error: listen EADDRINUSE: address already in use 0.0.0.0:4567 {"code":"EADDRINUSE","errno":"EADDRINUSE","syscall":"listen","address":"0.0.0.0","port":4567}
>2020-01-15T23:08:48.754Z [4567/1345] - info: [app] Shutdown (SIGTERM/SIGINT) Initialised

I ran ps aux | grep node output:

root      1188  0.0  0.3  63476  3716 pts/0    S    23:05   0:00 su node
node      1189  0.0  0.4  23020  4972 pts/0    S    23:05   0:00 bash
node      1272  0.1  4.1 605244 41980 ?        Ssl  23:08   0:00 /usr/bin/node /home/node/nodebb/loader.js
node      1283  1.7 11.0 893076 111140 ?       Sl   23:08   0:05 /usr/bin/node /home/node/nodebb/app.js
node      1369  0.0  0.3  40096  3536 pts/0    R+   23:13   0:00 ps aux
node      1370  0.0  0.0  14856  1004 pts/0    S+   23:13   0:00 grep --color=auto node

I thought maybe I could kill a process but that doesn't seem to work - Any ideas folks?

Altnernativley I guess I could possibly dump the redis DB and reload an known working snapshot as a workaround, learning something new in the process... can this be done with redis?

PitaJ

EADDRINUSE means nodebb is already running

omega

@PitaJ said in node install borked itself or I think it did... maybe nginx, any hunting tips?:

EADDRINUSE means nodebb is already running

omega

Right I restored a few month older snapshot of this nodebb install that is borked, and while it worked at the time of the snapshot, it was a good snapshot point, but yet same result. Nada.

It's behaving the same way, No server found - but the server it is up, nginx, nodebb, reds, they're all up and running, Hmmm... I'm at a bit of a loss where to start teasing this out. I could try another snapshot and repurpose but I've used a historical snapshot of this install and still same problem... btw this is on Digitalocean but that been fine for this and other light nodeBB installs.

If anyone can point to the one thing to check first that might shine a light I would as always be truly grateful.

oplik0

If you just want to make it work - try changing the port number in config.json.

But it definitely seems like there are some leftovers from a running instance that make the system think 4567 port is taken.

You can check what process that might be using netstat command (or more specifically, netstat -tapen will show you tcp connections and processes that utilize them). And then try to kill it.

omega

OMG If I ever ask this question again. Please tell me to check that the domain is active!!!!!!!

julian

@omega How does the saying go...

It's always DNS... even when you think it can't be the DNS, it's DNS.

omega

Check the credit card!!!