Troubleshooting Forever and port 4567

julian

Using forever start app.js, the NodeBB output is captured into forever's logging system. Use forever start loader.js to send the output to /logs/output.log.

You can find the forever log files in ~/.forever/

finid

@julian

Ok, I'll take a look at it and see if it will tell me anything useful.

finid

@julian

Ok, I found the log files and I think the contents point to something very wrong. I compared the contents of the log files in my local installation that's working to the production installation that's not:

Here's the output from a log file in the local install:

info: Initializing NodeBB v0.3.2
info: * using configuration stored in: /var/www/forum/config.json
info: * using redis store at 127.0.0.1:6379
info: * using themes stored in: /var/www/forum/node_modules
info: [user.startJobs] Registering User Jobs
warn: [plugins/nodebb-plugin-markdown] This plugin may not be compatible with your version of NodeBB. This may cause unintended behaviour or crashing.
warn: [plugins/nodebb-plugin-mentions] This plugin may not be compatible with your version of NodeBB. This may cause unintended behaviour or crashing.
warn: [plugins/nodebb-widget-essentials] staticDir is deprecated, use staticDirs instead
warn: [plugins/nodebb-theme-lavender] staticDir is deprecated, use staticDirs instead
warn: [plugins] filter:server.create_routes is deprecated and will maintain limited functionality until 0.4x
warn: [plugins] filter:admin.create_routes is deprecated and will maintain limited functionality until 0.4x
info: [themes] Compiling templates
info: [themes] Successfully compiled templates.
info: NodeBB attempting to listen on: 0.0.0.0:4567
info: NodeBB Ready

The last 2 lines give an indication that NodeBB is listening on it's default port, though it only says that it is attempting to listen on: 0.0.0.0:4567, not that it is actually listening. That's the subject of another topic, so I'll let it be for now.

Now, here's the contents of the production Forever log file.

[plugins] filter:server.create_routes is deprecated and will maintain limited functionality until 0.4x
warn: [plugins/nodebb-plugin-sso-github] "callbacked" deprecated as of 0.4x. Use asynchronous method instead for hook: filter:auth.init
error: [app] Encountered Uncaught Exception: meta is not defined
ReferenceError: meta is not defined
    at Object.GitHub.getStrategy (/forum/node_modules/nodebb-plugin-sso-github/library.js:22:7)
    at /forum/src/plugins.js:329:32
    at /forum/node_modules/async/lib/async.js:249:13
    at iterate (/forum/node_modules/async/lib/async.js:134:13)
    at Object.async.eachSeries (/forum/node_modules/async/lib/async.js:150:9)
    at Object.async.reduce (/forum/node_modules/async/lib/async.js:248:15)
    at Object.Plugins.fireHook (/forum/src/plugins.js:316:12)
    at /forum/src/routes/authentication.js:117:12
    at Object.Plugins.ready (/forum/src/plugins.js:57:4)
    at Object.Auth.createRoutes (/forum/src/routes/authentication.js:116:11)
error: [app] Could not restart server. Shutting down.
info: [app] Shutdown (SIGTERM/SIGINT) Initialised.
info: [app] Database connection closed.
info: [app] Shutdown complete.
error: Forever detected script was killed by signal: null
error: Forever restarting script for 223 time
info: NodeBB v0.3.2 Copyright (C) 2013 DesignCreatePlay Inc.
info: This program comes with ABSOLUTELY NO WARRANTY.
info: This is free software, and you are welcome to redistribute it under certain conditions.
info: 
info: Time: Thu Mar 20 2014 13:48:02 GMT-0500 (CDT)
info: Initializing NodeBB v0.3.2
info: * using configuration stored in: /forum/config.json
info: * using redis store at 127.0.0.1:6379
info: * using themes stored in: /forum/node_modules

Notice that there's no entry about compiling templates or starting NodeBB at the default ports.

Does this output tell you anything? Why does the script terminate at the point just after starting Redis?

julian

Looks like your github plugin is out of date. If you're on latest NodeBB, please update it to use the latest version. The accidental global meta was fixed a month ago at least

finid

@julian

I have a bunch of plugins installed. Is there a single command that will update all installed plugins at the same time?

julian

Sure thing: ./nodebb upgrade

finid

@julian

Well, before starting NodeBB with Forever, I git-pulled and upgraded, so that should have taken care of that. Obviously, it did not.

julian

~~... son of a b-... I swore I fixed it along with fb/tw/g+. Give me one second.~~ Turns out psychobunny wrote the original GH SSO, and it is up to date...

What version of the plugin are you running? npm ls nodebb-plugin-sso-github

finid

@julian

Output from ./nodebb upgrade

npm ERR! node -v v0.10.26
npm ERR! npm -v 1.4.3
npm ERR! code E404
npm http 304 https://registry.npmjs.org/nodebb-plugin-markdown
npm http 304 https://registry.npmjs.org/nodebb-widget-essentials
npm http 304 https://registry.npmjs.org/nodebb-plugin-color
npm http 304 https://registry.npmjs.org/nodebb-plugin-dailymotion
npm http 304 https://registry.npmjs.org/nodebb-theme-lavender
npm http 304 https://registry.npmjs.org/nodebb-plugin-emoji-extended
npm http 304 https://registry.npmjs.org/nodebb-plugin-42
npm http 304 https://registry.npmjs.org/nodebb-plugin-finder
npm http 304 https://registry.npmjs.org/nodebb-plugin-emailer-local
npm http 304 https://registry.npmjs.org/nodebb-plugin-openid-steam
npm http 304 https://registry.npmjs.org/nodebb-plugin-sketchfab
npm http 304 https://registry.npmjs.org/nodebb-plugin-linkcheck
npm http 304 https://registry.npmjs.org/nodebb-plugin-sso-github
npm http 304 https://registry.npmjs.org/nodebb-theme-vanilla
npm http 304 https://registry.npmjs.org/nodebb-plugin-sso-facebook
npm http 304 https://registry.npmjs.org/nodebb-plugin-sso-google
npm http 304 https://registry.npmjs.org/nodebb-plugin-vimeo
npm http 304 https://registry.npmjs.org/nodebb-plugin-twitter
npm http 304 https://registry.npmjs.org/nodebb-plugin-youtube
npm http 304 https://registry.npmjs.org/nodebb-plugin-sso-twitter
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /forum/npm-debug.log
npm ERR! not ok code 0

finid

An outdated plugin shouldn't really stop the entire application from starting.

finid

npm ls nodebb-plugin-sso-github

@julian

[email protected] /forum
└── [email protected]  extraneous

julian

npm install [email protected]

finid

@julian

Done. What next? Another ./nodebb upgrade or just run forever start app.js again?

julian

Give it another run through (./nodebb start && ./nodebb log), and let's see what happens. If you want to stop NodeBB after, ./nodebb stop

finid

Something new and maybe related. Never saw before.

make: Entering directory `forum/node_modules/hiredis/build'
  CC(target) Release/obj.target/hiredis/deps/hiredis/hiredis.o
  CC(target) Release/obj.target/hiredis/deps/hiredis/net.o
  CC(target) Release/obj.target/hiredis/deps/hiredis/sds.o
  CC(target) Release/obj.target/hiredis/deps/hiredis/async.o
  AR(target) Release/obj.target/deps/hiredis.a
  COPY Release/hiredis.a
  CXX(target) Release/obj.target/hiredis/src/hiredis.o
make: g++: Command not found
make: *** [Release/obj.target/hiredis/src/hiredis.o] Error 127
make: Leaving directory `forum/node_modules/hiredis/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:267:23)
gyp ERR! stack     at ChildProcess.EventEmitter.emit (events.js:98:17)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:797:12)
gyp ERR! System Linux 3.11.0-12-generic
gyp ERR! command "node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd forum/node_modules/hiredis
gyp ERR! node -v v0.10.26
gyp ERR! node-gyp -v v0.12.2
gyp ERR! not ok 
npm WARN optional dep failed, continuing [email protected]
npm WARN package.json [email protected] No repository field.
npm WARN package.json [email protected] No repository field.
npm WARN package.json [email protected] No repository field.
npm WARN package.json [email protected] No repository field.

Significant, right?

After installing g++ using apt-get install g++ and re-running ./nodebb upgrade, that error was not repeated.

finid

@julian

That triggers non-stop logging to the console. How can I send it to a file?

finid

No change. Something's very wrong.

info: NodeBB v0.3.2 Copyright (C) 2013 DesignCreatePlay Inc.
info: This program comes with ABSOLUTELY NO WARRANTY.
info: This is free software, and you are welcome to redistribute it under certain conditions.
info: 
info: Time: Thu Mar 20 2014 15:00:00 GMT-0500 (CDT)
info: Initializing NodeBB v0.3.2
info: * using configuration stored in: forum/config.json
info: * using redis store at 127.0.0.1:6379
info: * using themes stored in: forum/node_modules
info: [user.startJobs] Registering User Jobs
warn: [plugins/nodebb-widget-essentials] staticDir is deprecated, use staticDirs instead
warn: [plugins/nodebb-plugin-youtube] staticDir is deprecated, use staticDirs instead
warn: [plugins/nodebb-plugin-sso-github] staticDir is deprecated, use staticDirs instead
warn: [plugins/nodebb-plugin-sso-facebook] staticDir is deprecated, use staticDirs instead
warn: [plugins/nodebb-plugin-linkcheck] staticDir is deprecated, use staticDirs instead
warn: [plugins/nodebb-plugin-markdown] This plugin may not be compatible with your version of NodeBB. This may cause unintended behaviour or crashing.
warn: [plugins/nodebb-plugin-vimeo] staticDir is deprecated, use staticDirs instead
warn: [plugins/nodebb-plugin-dailymotion] staticDir is deprecated, use staticDirs instead
warn: [plugins/nodebb-plugin-sso-google] staticDir is deprecated, use staticDirs instead
warn: [plugins/nodebb-plugin-mentions] This plugin may not be compatible with your version of NodeBB. This may cause unintended behaviour or crashing.
warn: [plugins/nodebb-plugin-sso-twitter] staticDir is deprecated, use staticDirs instead
warn: [plugins/nodebb-plugin-sketchfab] staticDir is deprecated, use staticDirs instead
warn: [plugins/nodebb-plugin-openid-steam] staticDir is deprecated, use staticDirs instead
warn: [plugins/nodebb-plugin-42] staticDir is deprecated, use staticDirs instead
warn: [plugins/nodebb-theme-lavender] staticDir is deprecated, use staticDirs instead
warn: [plugins/nodebb-plugin-twitter] "callbacked" deprecated as of 0.4x. Use asynchronous method instead for hook: filter:scripts.get
warn: [plugins/nodebb-plugin-emoji-extended] "callbacked" deprecated as of 0.4x. Use asynchronous method instead for hook: filter:scripts.get
warn: [plugins/nodebb-plugin-linkcheck] "callbacked" deprecated as of 0.4x. Use asynchronous method instead for hook: filter:scripts.get
warn: [plugins/nodebb-plugin-42] "callbacked" deprecated as of 0.4x. Use asynchronous method instead for hook: filter:scripts.get
warn: [meta.scripts.get] Could not resolve mapped path: plugins/twitter/main.js. Are you sure it is defined by a plugin?
warn: [meta.scripts.get] Could not resolve mapped path: plugins/emoji-extended-js/lib/jquery.textcomplete.min.js. Are you sure it is defined by a plugin?
warn: [meta.scripts.get] Could not resolve mapped path: plugins/emoji-extended-js/emoji-textcomplete.js. Are you sure it is defined by a plugin?
warn: [meta.scripts.get (nodebb-plugin-linkcheck)] filter:scripts.get is deprecated, consider using "scripts" in plugin.json
warn: [meta.scripts.get (nodebb-plugin-42)] filter:scripts.get is deprecated, consider using "scripts" in plugin.json
warn: [plugins] filter:server.create_routes is deprecated and will maintain limited functionality until 0.4x
warn: [plugins/nodebb-plugin-sso-github] "callbacked" deprecated as of 0.4x. Use asynchronous method instead for hook: filter:auth.init
warn: [plugins/nodebb-plugin-sso-facebook] "callbacked" deprecated as of 0.4x. Use asynchronous method instead for hook: filter:auth.init
ReferenceError: meta is not defined
    at Object.Facebook.getStrategy (forum/node_modules/nodebb-plugin-sso-facebook/library.js:23:7)
    at forum/src/plugins.js:329:32
    at forum/node_modules/async/lib/async.js:249:13
    at iterate (forum/node_modules/async/lib/async.js:134:13)
    at forum/node_modules/async/lib/async.js:145:25
    at forum/node_modules/async/lib/async.js:251:17
    at forum/src/plugins.js:330:9
    at forum/node_modules/async/lib/async.js:249:13
    at iterate (forum/node_modules/async/lib/async.js:134:13)
    at Object.async.eachSeries (forum/node_modules/async/lib/async.js:150:9)
info: [app] Restarting...
info: [app] Shutdown (SIGTERM/SIGINT) Initialised.
info: [app] Database connection closed.
info: [app] Shutdown complete.
info: NodeBB v0.3.2 Copyright (C) 2013 DesignCreatePlay Inc.
info: This program comes with ABSOLUTELY NO WARRANTY.
info: This is free software, and you are welcome to redistribute it under certain conditions.
info: 
info: Time: Thu Mar 20 2014 15:00:01 GMT-0500 (CDT)
info: Initializing NodeBB v0.3.2
info: * using configuration stored in: forum/config.json
info: * using redis store at 127.0.0.1:6379
info: * using themes stored in: forum/node_modules

Just restarting non-stop. Had to kill it.

NodeBB itself is never started. That's why the port is not open and that's why I get the error message below when I attempt to access it:

[error] 9921#0: *148 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: site.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:4567/", host: "site.com"

It's like trying to connect to ssh on a server when the ssh serer is not running.

julian

Again, looks like this time the Facebook plugin is out of date. What registry are you using? The EU registry is occasionally behind...

finid

@julian

I have no idea what Registry I'm using. By this, I think all the plugins must be outdated. How do I change the registry and update each plugin individually, since ./nodebb ugrade does not seem to be doing it?

finid

I think this explains why the local installation, which does not have any plugins installed other than the bundled Markdown plugin, is working fine, even though I'm using a version of Nginx that does not have good support for WebSockets.

Which brings up a point I've made earlier. With about 2 dozen plugins already available, it's going to be a nightmare if an outdated plugin will prevent NodeBB from starting. I made the mistake of upgrading what's supposed to be my production server to Edge, but I did it because I wanted to start accessing it from a subfolder.

I hope all the plugin will be compatible with 0.4.0 when it is released.

For the moment, I'm stuck. I can't start NodeBB. Now, can I downgrade to stable 0.3.2? At least with that, I can access it by appending the port to the URL.