Forum is not available anymore

[baris]

The files in node_modules could change if someone installs a plugin in the ACP. Does this affect all routes? Does loading something like forum.com/api/config work?

What active plugins do you have?

[PitaJ]

@nicolas huh? Mutating source files? I'm talking about cosmic rays flipping bits here.

[Nicolas]

@baris /api/config - does not work.

$ curl localhost:8081/api/config
Cannot GET /api/config

Plugins. logs from Dev mode:

4/3 00:42 [18479] - info: [themes] Using theme nodebb-theme-overseers
4/3 00:42 [18479] - verbose: [plugins] Initializing plugins system
4/3 00:42 [18479] - verbose: [plugins] Found 5 js file(s) for plugin nodebb-theme-overseers
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-theme-overseers
4/3 00:42 [18479] - verbose: [plugins] Found 1 css file(s) for plugin nodebb-widget-essentials
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-widget-essentials
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-soundpack-default
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-mentions
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-mentions
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-mentions
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-markdown
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-markdown
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-markdown
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-markdown
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-emoji-apple
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-emoji-extended
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-emoji-extended
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-emoji-extended
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-composer-default
4/3 00:42 [18479] - verbose: [plugins] Found 11 js file(s) for plugin nodebb-plugin-composer-default
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-composer-default
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-composer-default
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-api
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-ns-likes
4/3 00:42 [18479] - verbose: [plugins] Found 2 js file(s) for plugin nodebb-plugin-ns-likes
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-likes
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-login
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-ns-points
4/3 00:42 [18479] - verbose: [plugins] Found 2 js file(s) for plugin nodebb-plugin-ns-points
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-points
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-slugify
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-write-api
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-write-api
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-write-api
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-plugin-ns-embed
4/3 00:42 [18479] - verbose: [plugins] Found 0 js file(s) for plugin nodebb-plugin-ns-embed
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-ns-embed
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-embed
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-dbsearch
4/3 00:42 [18479] - verbose: [plugins] Found 3 less file(s) for plugin nodebb-plugin-ns-custom-fields
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-ns-custom-fields
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-ns-custom-fields
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-custom-fields
4/3 00:42 [18479] - verbose: [plugins] Found 3 less file(s) for plugin nodebb-plugin-ns-awards
4/3 00:42 [18479] - verbose: [plugins] Found 0 js file(s) for plugin nodebb-plugin-ns-awards
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-ns-awards
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-ns-awards
4/3 00:42 [18479] - verbose: [plugins] Found 0 css file(s) for plugin nodebb-widget-ns-birthdays
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-widget-ns-birthdays
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-widget-ns-birthdays
4/3 00:42 [18479] - verbose: [plugins] Found 0 css file(s) for plugin nodebb-widget-ns-stats
4/3 00:42 [18479] - verbose: [plugins] Found 1 less file(s) for plugin nodebb-widget-ns-stats
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-widget-ns-stats
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-emailer-mailgun
4/3 00:42 [18479] - verbose: [plugins] Found 1 js file(s) for plugin nodebb-plugin-google-analytics
4/3 00:42 [18479] - verbose: [plugins] Loaded plugin: nodebb-plugin-google-analytics

4/3 00:42 [18479] - warn: [plugins/load] The following plugins may not be compatible with your version of NodeBB. This may cause unintended behaviour or crashing. In the event of an unresponsive NodeBB caused by this plugin, run `./nodebb reset -p PLUGINNAME` to disable it.
  * nodebb-plugin-ns-points
  * nodebb-plugin-emailer-mailgun

4/3 00:42 [18479] - info: [plugins] Plugins OK

[Nicolas]

/api/config in Dev Mode - works

[psychobunny]

@nicolas though I don't want to immediately suggest you upgrade from NodeBB v1.0.2 (that's a LONG time ago! haha... welcome back btw!)

I wonder if you ran npm i etc and that upgraded you to packages that are no longer compatible with that version of core

[Nicolas]

I did not do npm install, I deactivated at this point 2 plugins that could depend on external services like Captcha and Twitch Monitor.

Here is an example of Express, if I would install node modules likely express would update also:

[Nicolas]

Any ideas why ./nodebb start does not work, and ./nodebb dev works?
Anything I could do to rebuild something in a less harmful way to try ./nodebb start?
Any suggestions why Express does not work in production mode?

[Nicolas]

So I'm debugging Express, in production, Route registration does not happen. For example, in debug mode I see logs like:

  express:router:route get /user/:userslug/edit/password +0ms
  express:router:layer new / +0ms
  express:router:route get /user/:userslug/edit/password +0ms
  express:router:layer new / +0ms
  express:router:route get /user/:userslug/edit/password +0ms
  express:router:layer new / +0ms
  express:router:route new /api/user/:userslug/edit/password +0ms
  express:router:layer new /api/user/:userslug/edit/password +0ms
  express:router:route get /api/user/:userslug/edit/password +0ms
  express:router:layer new / +0ms
  express:router:route get /api/user/:userslug/edit/password +0ms
  express:router:layer new / +0ms
  express:router:route get /api/user/:userslug/edit/password +0ms
  express:router:layer new / +0ms
  express:router:route get /api/user/:userslug/edit/password +0ms
  express:router:layer new / +0ms
  express:router:route get /api/user/:userslug/edit/password +0ms
  express:router:layer new / +0ms
  express:router:route new /user/:userslug/settings +0ms
  express:router:layer new /user/:userslug/settings +0ms
  express:router:route get /user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route get /user/:userslug/settings +1ms
  express:router:layer new / +0ms
  express:router:route get /user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route get /user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route get /user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route get /user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route get /user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route new /api/user/:userslug/settings +0ms
  express:router:layer new /api/user/:userslug/settings +0ms
  express:router:route get /api/user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route get /api/user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route get /api/user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route get /api/user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route get /api/user/:userslug/settings +0ms
  express:router:layer new / +0ms
  express:router:route new /user/:userslug/session/:uuid +0ms
  express:router:layer new /user/:userslug/session/:uuid +0ms
  express:router:route delete /user/:userslug/session/:uuid +0ms

But in production mode I can see only initial Express operations:

express:application set "x-powered-by" to true +0ms
  express:application set "etag" to 'weak' +1ms
  express:application set "etag fn" to [Function: wetag] +1ms
  express:application set "env" to 'development' +0ms
  express:application set "query parser" to 'extended' +0ms
  express:application set "query parser fn" to [Function: parseExtendedQueryString] +0ms
  express:application set "subdomain offset" to 2 +0ms
  express:application set "trust proxy" to false +0ms
  express:application set "trust proxy fn" to [Function: trustNone] +1ms
  express:application booting in development mode +0ms
  express:application set "view" to [Function: View] +0ms
  express:application set "views" to '/web/overseers/node/forum/views' +0ms
  express:application set "jsonp callback name" to 'callback' +0ms
  express:application set "view engine" to 'tpl' +85ms
  express:application set "views" to '/web/overseers/node/forum/public/templates' +1ms
  express:application set "json spaces" to 0 +0ms
  express:router use / query +0ms
  express:router:layer new / +1ms
  express:router use / expressInit +0ms
  express:router:layer new / +0ms
  express:router use / <anonymous> +0ms
  express:router:layer new / +0ms
  express:application set "view cache" to true +1ms
  express:router use / compression +0ms
  express:router:layer new / +0ms
  express:router use  favicon +0ms
  express:router:layer new  +0ms
  express:router use /apple-touch-icon <anonymous> +0ms
  express:router:layer new /apple-touch-icon +0ms
  express:router use / urlencodedParser +26ms
  express:router:layer new / +1ms
  express:router use / jsonParser +0ms
  express:router:layer new / +0ms
  express:router use / cookieParser +0ms
  express:router:layer new / +0ms
  express:router use / <anonymous> +1ms
  express:router:layer new / +0ms
  express:router use / session +0ms
  express:router:layer new / +0ms
  express:router use / <anonymous> +0ms
  express:router:layer new / +0ms
  express:router use / <anonymous> +1ms
  express:router:layer new / +0ms
  express:router use / initialize +0ms
  express:router:layer new / +0ms
  express:router use / authenticate +0ms
  express:router:layer new / +0ms
  express:router use / <anonymous> +0ms
  express:router:layer new / +0ms
  express:router use / <anonymous> +1ms
  express:router:layer new / +0ms
  express:application set "cache" to true +1ms
  express:application set "minification" to true +0ms

That is why even the basic root route does not work since none of them are registered. Any hints where to look next?

[Nicolas]

@psychobunny @baris what am I risking by running forum as NODE_ENV=development ./nodebb start?

[PitaJ]

My guess is corrupted files. You can try setting up a new nodebb instance with a copy of your database.

Since it's affected by node_env it could be a nodebb file that was corrupted. Have you tried using git to detect a file diff?

Running with an ancient version of NodeBB is probably more dangerous than running in development mode. Might make MITM a little more likely.

[baris]

Can you post the output of git status as well?

You can always add console.logs into src/routes.js and webserver js to debug.

[PitaJ]

You might also want to try git diff --name-status HEAD^ as git status uses file size and modification time.

[Nicolas]

@baris Git Status

$ git status
On branch v1.x.x
Your branch is up-to-date with 'origin/v1.x.x'.
nothing to commit, working directory clean

@PitaJ Git DIff is package.json only. Likely because I have plugins.

[baris]

I am not sure if 1.0.2 has ./nodebb plugins command but if it has run that and paste the output. If it doesn't have that command you can run the following in mongodb cli db.objects.find({_key: "plugins:active"});

Also put a console.log here and check if it is triggered in dev and ./nodebb start.

[Nicolas]

@baris plugins command output:

$ ./nodebb plugins
5/3 04:06 [30726] - info: [database] Checking database indices.
5/3 04:06 [30726] - info: Active plugins:
	 - nodebb-theme-overseers
	 - nodebb-widget-essentials
	 - nodebb-plugin-soundpack-default
	 - nodebb-plugin-mentions
	 - nodebb-plugin-markdown
	 - nodebb-plugin-emoji-apple
	 - nodebb-plugin-emoji-extended
	 - nodebb-plugin-composer-default
	 - nodebb-plugin-ns-api
	 - nodebb-plugin-ns-likes
	 - nodebb-plugin-ns-login
	 - nodebb-plugin-ns-points
	 - nodebb-plugin-ns-slugify
	 - nodebb-plugin-write-api
	 - nodebb-plugin-ns-embed
	 - nodebb-plugin-dbsearch
	 - nodebb-plugin-ns-custom-fields
	 - nodebb-plugin-ns-awards
	 - nodebb-widget-ns-birthdays
	 - nodebb-widget-ns-stats
	 - nodebb-plugin-emailer-mailgun
	 - nodebb-plugin-google-analytics

Interesting, any of these plugins interact with external services maybe? That is why it's stopped working. If no changes to local plugins like my ("NS") or others, it should still work without any issues.

As for console.log, it's triggered for ./nodebb dev, but it is not triggered for ./nodebb start

[baris]

@nicolas Ok if it's not triggering for ./nodebb start you will have to add some more console.logs in here basically add a console.log for each block in that async.waterfall(). My guess is one of the steps before routes(app, middleware); is breaking in production mode, like a build step or a plugin that uses static:app.preload. Change the code to look like this .

async.waterfall([
	async.apply(cacheStaticFiles),
	async.apply(meta.themes.setupPaths),
	function(next) {
		console.log('1');
		plugins.init(app, middleware, next);
	},
	function(next) {
		console.log('2');
		async.parallel([
			async.apply(meta.templates.compile),
			async.apply(!skipJS ? meta.js.minify : meta.js.getFromFile, 'nodebb.min.js'),
			async.apply(!skipJS ? meta.js.minify : meta.js.getFromFile, 'acp.min.js'),
			async.apply(!skipLess ? meta.css.minify : meta.css.getFromFile),
			async.apply(meta.sounds.init),
			async.apply(meta.blacklist.load)
		], next);
	},
	function(results, next) {
		console.log('3');
		plugins.fireHook('static:app.preload', {
			app: app,
			middleware: middleware
		}, next);
	},
	function(next) {
		console.log('4');
		routes(app, middleware);
		next();
	}
], callback);

Run it in prod mode and let me know the output.

[Nicolas]

I feel we are very close. @baris Thank you for looking into this!

It chokes here - https://github.com/NodeBB/NodeBB/blob/v1.0.2/src/webserver.js#L92-L101

I do not see console.log in the next waterfall step after template compilation.

[PitaJ]

@nicolas after disabling all plugins so you still see this issue? That will quickly rule out a plugin issue

[baris]

@nicolas What logs did you get did it not print 3 ?

[Nicolas]

@baris It did not print 3