• Plugin & Theme Dev

    @baris plugins command output:

    $ ./nodebb plugins
    5/3 04:06 [30726] - info: [database] Checking database indices.
    5/3 04:06 [30726] - info: Active plugins:
    	 - nodebb-theme-overseers
    	 - nodebb-widget-essentials
    	 - nodebb-plugin-soundpack-default
    	 - nodebb-plugin-mentions
    	 - nodebb-plugin-markdown
    	 - nodebb-plugin-emoji-apple
    	 - nodebb-plugin-emoji-extended
    	 - nodebb-plugin-composer-default
    	 - nodebb-plugin-ns-api
    	 - nodebb-plugin-ns-likes
    	 - nodebb-plugin-ns-login
    	 - nodebb-plugin-ns-points
    	 - nodebb-plugin-ns-slugify
    	 - nodebb-plugin-write-api
    	 - nodebb-plugin-ns-embed
    	 - nodebb-plugin-dbsearch
    	 - nodebb-plugin-ns-custom-fields
    	 - nodebb-plugin-ns-awards
    	 - nodebb-widget-ns-birthdays
    	 - nodebb-widget-ns-stats
    	 - nodebb-plugin-emailer-mailgun
    	 - nodebb-plugin-google-analytics
    

    Interesting, any of these plugins interact with external services maybe? That is why it's stopped working. If no changes to local plugins like my ("NS") or others, it should still work without any issues.

    As for console.log, it's triggered for ./nodebb dev, but it is not triggered for ./nodebb start

  • NodeBB

    @nicolas Ok if it's not triggering for ./nodebb start you will have to add some more console.logs in here basically add a console.log for each block in that async.waterfall(). My guess is one of the steps before routes(app, middleware); is breaking in production mode, like a build step or a plugin that uses static:app.preload. Change the code to look like this .

    async.waterfall([
    	async.apply(cacheStaticFiles),
    	async.apply(meta.themes.setupPaths),
    	function(next) {
    		console.log('1');
    		plugins.init(app, middleware, next);
    	},
    	function(next) {
    		console.log('2');
    		async.parallel([
    			async.apply(meta.templates.compile),
    			async.apply(!skipJS ? meta.js.minify : meta.js.getFromFile, 'nodebb.min.js'),
    			async.apply(!skipJS ? meta.js.minify : meta.js.getFromFile, 'acp.min.js'),
    			async.apply(!skipLess ? meta.css.minify : meta.css.getFromFile),
    			async.apply(meta.sounds.init),
    			async.apply(meta.blacklist.load)
    		], next);
    	},
    	function(results, next) {
    		console.log('3');
    		plugins.fireHook('static:app.preload', {
    			app: app,
    			middleware: middleware
    		}, next);
    	},
    	function(next) {
    		console.log('4');
    		routes(app, middleware);
    		next();
    	}
    ], callback);
    

    Run it in prod mode and let me know the output.

  • Plugin & Theme Dev

    I feel we are very close. @baris Thank you for looking into this!

    It chokes here - https://github.com/NodeBB/NodeBB/blob/v1.0.2/src/webserver.js#L92-L101

    I do not see console.log in the next waterfall step after template compilation.

  • Global Moderator Plugin & Theme Dev

    @nicolas after disabling all plugins so you still see this issue? That will quickly rule out a plugin issue

  • NodeBB

    @nicolas What logs did you get did it not print 3 ?

  • Plugin & Theme Dev

    @baris It did not print 3

  • NodeBB

    Ok now modify it again to the below to figure out which of those steps is breaking. Like below:

    async.waterfall([
    	async.apply(cacheStaticFiles),
    	async.apply(meta.themes.setupPaths),
    	function(next) {
    		console.log('1');
    		plugins.init(app, middleware, next);
    	},
    	function(next) {
    		console.log('2');
    		async.waterfall([
    			function (next) {
    				meta.templates.compile(function (err) {
    					console.log('2.compile');
    					next(err);
    				});
    			},
    			function (next) {
    				if (!skipJS) {
    					meta.js.minify('nodebb.min.js', function (err) {
    						console.log('2.b.minify');
    						next(err);
    					})
    				} else {
    					meta.js.getFromFile('nodebb.min.js', function (err) {
    						console.log('2.b.getFromFile');
    						next(err);
    					});
    				}
    			},
    			function (next) {
    				if (!skipJS) {
    					meta.js.minify('acp.min.js', function (err) {
    						console.log('2.b.minifyjs');
    						next(err);
    					})
    				} else {
    					meta.js.getFromFile('acp.min.js', function (err) {
    						console.log('2.b.getFromFilejs');
    						next(err);
    					});
    				}
    			},
    			function (next) {
    				if (!skipLess) {
    					meta.css.minify(function (err) {
    						console.log('2.b.minifycss');
    						next(err);
    					})
    				} else {
    					meta.css.getFromFile(function (err) {
    						console.log('2.b.getFromFileCss');
    						next(err);
    					});
    				}
    			},
    			function (next) {
    				meta.sounds.init(function (err) {
    					console.log('2.sounds');
    					next(err);
    				})
    			},
    			function (next) {
    				meta.sounds.init(function (err) {
    					console.log('2.blaclist');
    					next(err);
    				})
    			}
    		], next);
    	},
    	function(results, next) {
    		console.log('3');
    		plugins.fireHook('static:app.preload', {
    			app: app,
    			middleware: middleware
    		}, next);
    	},
    	function(next) {
    		console.log('4');
    		routes(app, middleware);
    		next();
    	}
    ], callback);
    

    Sorry it's a bit long due to all the callbacks and error logging.

  • Plugin & Theme Dev

    A bit weird results. You have switched to waterfall, now it goes to the next step, but chokes there.

    1
    2
    2.compile
    2.b.minify
    2.b.minifyjs
    5/3 04:41 [31364] - info: NodeBB Ready
    5/3 04:41 [31364] - info: Enabling 'trust proxy'
    2.b.minifycss
    5/3 04:41 [31364] - info: NodeBB is now listening on: 0.0.0.0:8081
    2.sounds
    2.blaclist
    3
    

    Reverted back to parallel:

    1
    2
    
  • NodeBB

    If it goes to next step with waterfall, it could be due to the server not having enough ram to run all the build process in parallel.

    Keep it as a waterfall and take a look at your active plugins that are using the hook 'static:app.preload' one of them is not calling the callback properly or has an issue.

    You can run grep -rnw "static:app.preload" node_modules/ to search for that in the node_modules folder.

  • Plugin & Theme Dev

    I have switched your proposed extra flow with console logs to parallel, and receive these results:

    1
    2.sounds
    5/3 04:54 [31928] - error: [sounds] Could not initialise sounds: EEXIST: file already exists, symlink '/web/overseers/node/forum/public/uploads/sounds/.gitignore' -> '/web/overseers/node/forum/public/sounds/.gitignore'
    2.blaclist
    2.b.minifycss
    2.b.minify
    5/3 04:54 [31928] - info: NodeBB Ready
    5/3 04:54 [31928] - info: Enabling 'trust proxy'
    2.compile
    5/3 04:54 [31928] - info: NodeBB is now listening on: 0.0.0.0:8081
    
  • NodeBB

    Does it work in prod if you disable sounds with this?

    function (next) {
            next();
    	// meta.sounds.init(function (err) {
    	//	console.log('2.sounds');
    	//	next(err);
    	//})
    },
  • NodeBB

    Looks like sounds module is just logging the error but not preventing the flow.

    I suggest switching to async.waterfall and checking the plugins using 'static:app.preload' hook and making sure the server has enough memory.

  • Plugin & Theme Dev

    @baris yep. Sounds are not the problem, the result is the same with the disabled sound step.

    Interesting if memory is an issue now. Yes, it's a 512MB server, for some reason, it's a limit now after 5 years?..
    When I look at htop, it peaks at max memory when forum startups, but it was not an issue before.

    I can try to ask the provider to give more memory to check the hypothesis and let you know if it helped.

  • NodeBB

    Try to setup some swap if you can, and yes memory can be an issue over time if the database is on the same server as well.

    You said you hit step 3 so that means the build was succesfull with async.waterfall it means it is getting stuck in one of the plugins that is using 'static:app.preload' you can find which plugins use it with the above grep command.

  • Plugin & Theme Dev

    Thank you everybody for helping me with forum troubleshooting. The problem has been found. It's memory limit. After a few years of use, 512MB RAM + 256MB swap is not enough anymore (I have some suspicions that swap does not kick in, but it's a topic for a different story).

    If somebody searches, and no changes plugins, core, or hardware, and you do not have plugins that depend on external services - check if it's a memory issue.

Suggested Topics

| |