Forum is not available anymore

[Nicolas]

I feel we are very close. @baris Thank you for looking into this!

It chokes here - https://github.com/NodeBB/NodeBB/blob/v1.0.2/src/webserver.js#L92-L101

I do not see console.log in the next waterfall step after template compilation.

[PitaJ]

@nicolas after disabling all plugins so you still see this issue? That will quickly rule out a plugin issue

[baris]

@nicolas What logs did you get did it not print 3 ?

[Nicolas]

@baris It did not print 3

[baris]

Ok now modify it again to the below to figure out which of those steps is breaking. Like below:

async.waterfall([
	async.apply(cacheStaticFiles),
	async.apply(meta.themes.setupPaths),
	function(next) {
		console.log('1');
		plugins.init(app, middleware, next);
	},
	function(next) {
		console.log('2');
		async.waterfall([
			function (next) {
				meta.templates.compile(function (err) {
					console.log('2.compile');
					next(err);
				});
			},
			function (next) {
				if (!skipJS) {
					meta.js.minify('nodebb.min.js', function (err) {
						console.log('2.b.minify');
						next(err);
					})
				} else {
					meta.js.getFromFile('nodebb.min.js', function (err) {
						console.log('2.b.getFromFile');
						next(err);
					});
				}
			},
			function (next) {
				if (!skipJS) {
					meta.js.minify('acp.min.js', function (err) {
						console.log('2.b.minifyjs');
						next(err);
					})
				} else {
					meta.js.getFromFile('acp.min.js', function (err) {
						console.log('2.b.getFromFilejs');
						next(err);
					});
				}
			},
			function (next) {
				if (!skipLess) {
					meta.css.minify(function (err) {
						console.log('2.b.minifycss');
						next(err);
					})
				} else {
					meta.css.getFromFile(function (err) {
						console.log('2.b.getFromFileCss');
						next(err);
					});
				}
			},
			function (next) {
				meta.sounds.init(function (err) {
					console.log('2.sounds');
					next(err);
				})
			},
			function (next) {
				meta.sounds.init(function (err) {
					console.log('2.blaclist');
					next(err);
				})
			}
		], next);
	},
	function(results, next) {
		console.log('3');
		plugins.fireHook('static:app.preload', {
			app: app,
			middleware: middleware
		}, next);
	},
	function(next) {
		console.log('4');
		routes(app, middleware);
		next();
	}
], callback);

Sorry it's a bit long due to all the callbacks and error logging.

[Nicolas]

A bit weird results. You have switched to waterfall, now it goes to the next step, but chokes there.

1
2
2.compile
2.b.minify
2.b.minifyjs
5/3 04:41 [31364] - info: NodeBB Ready
5/3 04:41 [31364] - info: Enabling 'trust proxy'
2.b.minifycss
5/3 04:41 [31364] - info: NodeBB is now listening on: 0.0.0.0:8081
2.sounds
2.blaclist
3

Reverted back to parallel:

1
2

[baris]

If it goes to next step with waterfall, it could be due to the server not having enough ram to run all the build process in parallel.

Keep it as a waterfall and take a look at your active plugins that are using the hook 'static:app.preload' one of them is not calling the callback properly or has an issue.

You can run grep -rnw "static:app.preload" node_modules/ to search for that in the node_modules folder.

[Nicolas]

I have switched your proposed extra flow with console logs to parallel, and receive these results:

1
2.sounds
5/3 04:54 [31928] - error: [sounds] Could not initialise sounds: EEXIST: file already exists, symlink '/web/overseers/node/forum/public/uploads/sounds/.gitignore' -> '/web/overseers/node/forum/public/sounds/.gitignore'
2.blaclist
2.b.minifycss
2.b.minify
5/3 04:54 [31928] - info: NodeBB Ready
5/3 04:54 [31928] - info: Enabling 'trust proxy'
2.compile
5/3 04:54 [31928] - info: NodeBB is now listening on: 0.0.0.0:8081

[baris]

Does it work in prod if you disable sounds with this?

function (next) {
        next();
	// meta.sounds.init(function (err) {
	//	console.log('2.sounds');
	//	next(err);
	//})
},

[baris]

Looks like sounds module is just logging the error but not preventing the flow.

I suggest switching to async.waterfall and checking the plugins using 'static:app.preload' hook and making sure the server has enough memory.

[Nicolas]

@baris yep. Sounds are not the problem, the result is the same with the disabled sound step.

Interesting if memory is an issue now. Yes, it's a 512MB server, for some reason, it's a limit now after 5 years?..
When I look at htop, it peaks at max memory when forum startups, but it was not an issue before.

I can try to ask the provider to give more memory to check the hypothesis and let you know if it helped.

[baris]

Try to setup some swap if you can, and yes memory can be an issue over time if the database is on the same server as well.

You said you hit step 3 so that means the build was succesfull with async.waterfall it means it is getting stuck in one of the plugins that is using 'static:app.preload' you can find which plugins use it with the above grep command.

[Nicolas]

Thank you everybody for helping me with forum troubleshooting. The problem has been found. It's memory limit. After a few years of use, 512MB RAM + 256MB swap is not enough anymore (I have some suspicions that swap does not kick in, but it's a topic for a different story).

If somebody searches, and no changes plugins, core, or hardware, and you do not have plugins that depend on external services - check if it's a memory issue.