Weird nodebb 502 bad gateway

etakmit

I'm on a roll today.

Running nodebb and nginx to show my forum URL without the port. Was up and running just fine yesterday. I've installed a single plugin (emailer - mailgun). Earlier I posted a question here about that plugin (had an error sending mail). So it was suggested that I run the command ./nodebb dev. I did so from /opt/nodebb as root (after stopping the service properly)

Got through the problem above - email sends great now. So I kill the debug and restart nodebb via its service (sudo service nodebb start). Now I'm seeing 502 bad gateway from nginx.

Something isn't right and I'm just flat out missing it I think. The interesting part is I can stop nodebb and go run the debug instance as above and the site becomes reachable again.

config.json

"url": "http://127.0.0.1:4567", (it worked previously like this - I have tried it also as just 127.0.0.1:4567)
"secret": xxx
"database": "mongo",
"mongo": {
"host": "127.0.0.1",
"port": "27017",
"username": "nodebb",
"password": xxx
"database": "nodebb"

nginx forum.conf

listen 80;

server_name forum.com;

location / {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Host $http_host;
    proxy_set_header X-NginX-Proxy true;

    proxy_pass http://127.0.0.1:4567;
    proxy_redirect off;

    # Socket.IO Support
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";

Thoughts? The weirdest part is that it works when running debug but not when normal (and it was working entirely fine until I stopped it). When normal it does run nodebb as the nodebb user.

PitaJ

@etakmit the nginx config you pasted I think got cut off.

etakmit

This is the conf for my forum within sites-available. There's no more to it (and its actively working as long as I don't start nodebb via the upstart ubuntu script). Right now I'm running ./nodebb log and its working. but as soon as i start it via service nodebb start - it's no longer functional. time to go dig some more

server {
    listen 80;

    server_name nxsforum.com;

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;

        proxy_pass http://127.0.0.1:4567;
        proxy_redirect off;

        # Socket.IO Support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

PitaJ

The ./nodebb commands must be run within the nodebb directory, otherwise the cwd environment is wrong. I'd check to make sure you're cding into the nodebb directory before trying to ./nodebb start.

edit: also, make sure no node processes are running.

etakmit

@PitaJ I think you're not quite on the same page.

When i run ./nodebb <anything> my nodebb and nginx instance are just fine (get to the site, no problems).

BUT when I run

sudo service nodebb start

As per the file created following the install instructions, nodebb starts up fine and I see its processes running. BUT nginx no longer treats it properly and I get the 502 bad gateway. It's almost like its ignoring the config.json file on startup (which isn't the case or it'd never connect to the DB).

I followed the instructions on this page https://docs.nodebb.org/en/latest/installing/os/ubuntu.html and here's the section (and until stopping it the first time - everything worked fine)

The last setup item is to configure NodeBB to start automatically. Modern linux systems have adopted systemd as the default init system. Configure nodebb to start via a systemd unit file at the location /lib/systemd/system/nodebb.service:

[Unit]
Description=NodeBB forum for Node.js.
Documentation=http://nodebb.readthedocs.io/en/latest/
After=system.slice multi-user.target

[Service]
Type=simple
User=nodebb

StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=nodebb

Environment=NODE_ENV=production
WorkingDirectory=/opt/nodebb
ExecStart=/usr/bin/node loader.js --no-daemon --no-silent
Restart=always

[Install]
WantedBy=multi-user.target
Finally, enable and start NodeBB:

$ sudo systemctl enable nodebb
$ sudo service nodebb start
$ sudo service nodebb status

PitaJ

@etakmit

Can you try running cd /opt/nodebb && /usr/bin/node loader.js --no-daemon --no-silent?

etakmit

@PitaJ said in Weird nodebb 502 bad gateway:

/usr/bin/node loader.js --no-daemon --no-silent

Sure can - no problems. Spins up fine, loads fine, etc.

I notice though. Looking at a simple ps aux | grep -i node that nodebb is running as root in that case (I have to "sudo" it to start it up).

As soon as I run the sudo service nodebb start it shows up fine in ps aux EXCEPT that the user is nodebb (as it should be).

It feels very permission like when trying to figure this out.

PitaJ

@etakmit I have a feeling that the permissions of your nodebb directory are messed up. Make sure they're owned by the nodebb user.

etakmit

@PitaJ they are. As soon as I saw that I had the same thought. Unless I need lots of random write permissions things look correct.

drwxr-xr-x 11 nodebb nodebb 4096 Jan 24 14:41 nodebb

drwxr-xr-x  11 nodebb nodebb  4096 Jan 24 15:17 .
drwxr-xr-x   3 root   root    4096 Jan 24 14:31 ..
-rw-r--r--   1 nodebb nodebb 11477 Jan 23 16:15 app.js
-rw-r--r--   1 nodebb nodebb   665 Jan 23 16:15 bcrypt.js
-rw-r--r--   1 nodebb nodebb  2950 Jan 23 16:15 build.js
-rw-r--r--   1 nodebb nodebb   170 Jan 23 16:15 .codeclimate.yml
-rw-r--r--   1 nodebb nodebb   292 Jan 24 13:46 config.json
-rw-r--r--   1 nodebb nodebb   549 Jan 23 16:15 Dockerfile
-rw-r--r--   1 nodebb nodebb   157 Jan 23 16:15 .editorconfig
-rw-r--r--   1 nodebb nodebb   284 Jan 23 16:15 .eslintignore
-rw-r--r--   1 nodebb nodebb  3614 Jan 23 16:15 .eslintrc.json
drwxr-xr-x   8 nodebb nodebb  4096 Jan 23 16:15 .git
-rw-r--r--   1 nodebb nodebb   257 Jan 23 16:15 .gitattributes
drwxr-xr-x   2 nodebb nodebb  4096 Jan 23 16:15 .github
-rw-r--r--   1 nodebb nodebb   797 Jan 23 16:15 .gitignore
-rw-r--r--   1 nodebb nodebb  2645 Jan 23 16:15 Gruntfile.js
drwxr-xr-x   3 nodebb nodebb  4096 Jan 23 16:15 install
-rw-r--r--   1 nodebb nodebb   455 Jan 23 16:15 .jsbeautifyrc
-rw-r--r--   1 nodebb nodebb  5317 Jan 23 16:15 .jshintrc
-rw-r--r--   1 nodebb nodebb 35147 Jan 23 16:15 LICENSE
-rw-r--r--   1 nodebb nodebb  5516 Jan 23 16:15 loader.js
drwxr-xr-x   2 nodebb nodebb  4096 Jan 23 16:20 logs
-rw-r--r--   1 nodebb nodebb  1716 Jan 23 16:15 minifier.js
-rwxr-xr-x   1 nodebb nodebb 13058 Jan 23 16:15 nodebb
-rw-r--r--   1 nodebb nodebb    17 Jan 23 16:15 nodebb.bat
drwxr-xr-x 477 nodebb nodebb 20480 Jan 23 17:06 node_modules
-rw-r--r--   1 nodebb nodebb  3910 Jan 23 16:15 package.json
drwxr-xr-x  10 nodebb nodebb  4096 Jan 24 15:09 public
-rw-r--r--   1 nodebb nodebb  5429 Jan 23 16:15 README.md
drwxr-xr-x  21 nodebb nodebb  4096 Jan 23 16:15 src
drwxr-xr-x   5 nodebb nodebb  4096 Jan 23 16:15 test
-rw-r--r--   1 nodebb nodebb  2064 Jan 23 16:15 .travis.yml
drwxr-xr-x   2 nodebb nodebb  4096 Jan 23 16:15 .tx

PitaJ

@etakmit try sudo -u nodebb -s cd /opt/nodebb && /usr/bin/node loader.js --no-daemon --no-silent

etakmit

@PitaJ said in Weird nodebb 502 bad gateway:

sudo -u nodebb -s cd /opt/nodebb && /usr/bin/node loader.js --no-daemon --no-silent

Ooh much closer

cd /opt/nodebb && /usr/bin/node loader.js --no-daemon --no-silent

NodeBB v1.4.2 Copyright (C) 2013-2014 NodeBB Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it under certain conditions.
For the full license, please visit: http://www.gnu.org/copyleft/gpl.html

Clustering enabled: Spinning up 1 process(es).

24/1 15:26:54 [6138] - info: Time: Tue Jan 24 2017 15:26:54 GMT-0500 (EST)
24/1 15:26:54 [6138] - info: Initializing NodeBB v1.4.2
24/1 15:26:55 [6138] - info: initializing NodeBB ...
24/1 15:26:56 [6138] - info: Routes added
24/1 15:26:56 [6138] - error: Could not initialise sound files:EACCES: permission denied, unlink '/opt/nodebb/public/sounds/notification.mp3'

So what I take from this is that it never finishes starting up because of that error.

looking at the public/sounds folder

drwxr-xr-x  2 root   root     4096 Jan 24 15:09 sounds

I'm guessing this is it. changing ownership. will reply in a few with results.

etakmit

yep we have a winner.

super odd. somewhere in the running it directly as root it changed ownership of the sound files.

fixed/resolved.

WildPointer

Hi there,
I'm having the same problem and it just happened after a power outage.
When I invoke ./nodebb start nothing really happens but when I run ./nodebb dev or node loader.js --no-daemon --no-silent it works like a charm
it's also working fine with forever start app.js and the only problem seems to be the ./nodebb start.
p.s. logfiles are empty, all permissions are correct and nothing has really changed.
cheers

PitaJ

@WildPointer that's a totally different problem. What version of NodeBB do you have?

WildPointer

@PitaJ said in Weird nodebb 502 bad gateway:

at version of NodeBB do you have?

currently running 1.0.3
After making backups, I ran the ./nodebb upgrade which did some stuff but the problem was still there and I rollebacked and decided to run it via forever.
just wondering why it would not work the way that it did for 2 years