Error while restarting the forum after several plugin update

[crazycells]

@baris Thanks, it is possible. I believe we have deleted almost all of the plugins, but somehow got similar errors so decided to install 1.13.2 with a back up database from several hours back.

We have encountered very serious UI problems in our upgrade, all pages were blank and they were all coming when the page is refreshed but none of the buttons were clickable...

After installing 3 times and getting the same problem, we have found the solution here:

https://community.nodebb.org/topic/14672/fresh-installation-seems-broken

I think this needs to be fixed, so others don't have the same problems. We were worried that there was problem with our back up database. Currently, our forum is back on... and we have only lost several hours of posts, which was acceptable for us.

I will update the forum with bugs as we encounter them...

[crazycells]

@baris We have some problems, but I do not know what... first of all, normally restarting and rebuilding the forum takes less than 30 seconds, however currently it is taking more than 1 minutes...

Forum looks ok, but customize plugin does not seem to be working, both changes in .tpl files and changes in translations are not appearing in the forum...

Here is the log:

[32m  started
(node:11248) DeprecationWarning: The option `autoReconnect` is incompatible with the unified topology, please read more by visiting http://bit.ly/2D8WfT6
(node:11248) DeprecationWarning: The option `reconnectTries` is incompatible with the unified topology, please read more by visiting http://bit.ly/2D8WfT6
(node:11248) DeprecationWarning: The option `reconnectInterval` is incompatible with the unified topology, please read more by visiting http://bit.ly/2D8WfT6
2020-04-16T23:28:46.299Z [4568/11248] - info: [build] Building in series mode
2020-04-16T23:28:46.299Z [4568/11248] - info: [build]         plugin static dirs  build started
2020-04-16T23:28:46.330Z [4568/11248] - info: [build]         plugin static dirs  build completed in 0.031sec
2020-04-16T23:28:46.330Z [4568/11248] - info: [build]          requirejs modules  build started
2020-04-16T23:28:53.815Z [4568/11248] - info: [build]          requirejs modules  build completed in 7.485sec
2020-04-16T23:28:53.816Z [4568/11248] - info: [build]           client js bundle  build started
2020-04-16T23:28:57.982Z [4568/11248] - info: [build]           client js bundle  build completed in 4.166sec
2020-04-16T23:28:57.986Z [4568/11248] - info: [build]            admin js bundle  build started
2020-04-16T23:29:01.630Z [4568/11248] - info: [build]            admin js bundle  build completed in 3.644sec
2020-04-16T23:29:01.630Z [4568/11248] - info: [build]         client side styles  build started
2020-04-16T23:29:12.275Z [4568/11248] - info: [build]         client side styles  build completed in 10.645sec
2020-04-16T23:29:12.275Z [4568/11248] - info: [build] admin control panel styles  build started
2020-04-16T23:29:22.747Z [4568/11248] - info: [build] admin control panel styles  build completed in 10.472sec
2020-04-16T23:29:22.747Z [4568/11248] - info: [build]                  templates  build started
2020-04-16T23:29:32.754Z [4568/11248] - info: [build]                  templates  build completed in 10.007sec
2020-04-16T23:29:32.754Z [4568/11248] - info: [build]                  languages  build started
2020-04-16T23:29:46.563Z [4568/11248] - info: [build]                  languages  build completed in 13.809sec
2020-04-16T23:29:46.563Z [4568/11248] - info: [build]                     sounds  build started
2020-04-16T23:29:46.569Z [4568/11248] - info: [build]                     sounds  build completed in 0.006sec
2020-04-16T23:29:46.570Z [4568/11248] - info: [build] Asset compilation successful. Completed in 60.27sec.
[cluster] Restarting...
2020-04-16T23:29:46.602Z [4567/11247] - info: [app] Shutdown (SIGTERM/SIGINT) Initialised.
2020-04-16T23:29:46.607Z [4569/11249] - info: [app] Shutdown (SIGTERM/SIGINT) Initialised.
2020-04-16T23:29:46.613Z [4568/11248] - info: [app] Shutdown (SIGTERM/SIGINT) Initialised.
Clustering enabled: Spinning up 3 process(es).

2020-04-16T23:29:46.685Z [4568/11248] - info: [app] Web server closed to connections.
2020-04-16T23:29:46.684Z [4569/11249] - info: [app] Web server closed to connections.
2020-04-16T23:29:46.711Z [4568/11248] - info: [app] Live analytics saved.
2020-04-16T23:29:46.720Z [4569/11249] - info: [app] Live analytics saved.
2020-04-16T23:29:46.742Z [4567/11247] - info: [app] Web server closed to connections.
2020-04-16T23:29:46.786Z [4568/11248] - info: [app] Database connection closed.
2020-04-16T23:29:46.784Z [4567/11247] - info: [app] Live analytics saved.
2020-04-16T23:29:46.788Z [4568/11248] - info: [app] Shutdown complete.
2020-04-16T23:29:46.876Z [4569/11249] - info: [app] Database connection closed.
2020-04-16T23:29:46.876Z [4569/11249] - info: [app] Shutdown complete.
2020-04-16T23:29:46.945Z [4567/11247] - info: [app] Database connection closed.
2020-04-16T23:29:46.947Z [4567/11247] - info: [app] Shutdown complete.
[cluster] Child Process (11247) has exited (code: 0, signal: null)
[cluster] Child Process (11248) has exited (code: 0, signal: null)
[cluster] Child Process (11249) has exited (code: 0, signal: null)
2020-04-16T23:29:48.511Z [4567/13902] - info: Initializing NodeBB v1.13.2 https://XXXXXX
(node:13902) DeprecationWarning: The option `autoReconnect` is incompatible with the unified topology, please read more by visiting http://bit.ly/2D8WfT6
(node:13903) DeprecationWarning: The option `autoReconnect` is incompatible with the unified topology, please read more by visiting http://bit.ly/2D8WfT6
(node:13902) DeprecationWarning: The option `reconnectTries` is incompatible with the unified topology, please read more by visiting http://bit.ly/2D8WfT6
(node:13904) DeprecationWarning: The option `autoReconnect` is incompatible with the unified topology, please read more by visiting http://bit.ly/2D8WfT6
(node:13902) DeprecationWarning: The option `reconnectInterval` is incompatible with the unified topology, please read more by visiting http://bit.ly/2D8WfT6
(node:13904) DeprecationWarning: The option `reconnectTries` is incompatible with the unified topology, please read more by visiting http://bit.ly/2D8WfT6
(node:13904) DeprecationWarning: The option `reconnectInterval` is incompatible with the unified topology, please read more by visiting http://bit.ly/2D8WfT6
(node:13903) DeprecationWarning: The option `reconnectTries` is incompatible with the unified topology, please read more by visiting http://bit.ly/2D8WfT6
(node:13903) DeprecationWarning: The option `reconnectInterval` is incompatible with the unified topology, please read more by visiting http://bit.ly/2D8WfT6
2020-04-16T23:29:54.126Z [4568/13903] - info: [socket.io] Restricting access to origin: https://XXXXX
2020-04-16T23:29:54.132Z [4569/13904] - info: [socket.io] Restricting access to origin: https://XXXXX
2020-04-16T23:29:54.198Z [4567/13902] - info: [socket.io] Restricting access to origin: https://XXXXX
2020-04-16T23:29:55.380Z [4567/13902] - warn: [deprecated] requiring core modules with `module.parent.require('./database'
Require stack:
- /home/nodebb/forum/nodebb/src/plugins/index.js
- /home/nodebb/forum/nodebb/src/groups/index.js
- /home/nodebb/forum/nodebb/src/user/index.js
- /home/nodebb/forum/nodebb/src/events.js
- /home/nodebb/forum/nodebb/src/meta/themes.js
- /home/nodebb/forum/nodebb/src/meta/index.js
- /home/nodebb/forum/nodebb/src/start.js
- /home/nodebb/forum/nodebb/require-main.js
- /home/nodebb/forum/nodebb/app.js)` is deprecated. Please use `require.main.require("./src/<module_name>")` instead.
    at Object.<anonymous> (/home/nodebb/forum/nodebb/node_modules/nodebb-plugin-audio-embed/library.js:11:24)
2020-04-16T23:29:55.387Z [4569/13904] - warn: [deprecated] requiring core modules with `module.parent.require('./database'
Require stack:
- /home/nodebb/forum/nodebb/src/plugins/index.js
- /home/nodebb/forum/nodebb/src/groups/index.js
- /home/nodebb/forum/nodebb/src/user/index.js
- /home/nodebb/forum/nodebb/src/events.js
- /home/nodebb/forum/nodebb/src/meta/themes.js
- /home/nodebb/forum/nodebb/src/meta/index.js
- /home/nodebb/forum/nodebb/src/start.js
- /home/nodebb/forum/nodebb/require-main.js
- /home/nodebb/forum/nodebb/app.js)` is deprecated. Please use `require.main.require("./src/<module_name>")` instead.
    at Object.<anonymous> (/home/nodebb/forum/nodebb/node_modules/nodebb-plugin-audio-embed/library.js:11:24)
2020-04-16T23:29:55.388Z [4568/13903] - warn: [deprecated] requiring core modules with `module.parent.require('./database'
Require stack:
- /home/nodebb/forum/nodebb/src/plugins/index.js
- /home/nodebb/forum/nodebb/src/groups/index.js
- /home/nodebb/forum/nodebb/src/user/index.js
- /home/nodebb/forum/nodebb/src/events.js
- /home/nodebb/forum/nodebb/src/meta/themes.js
- /home/nodebb/forum/nodebb/src/meta/index.js
- /home/nodebb/forum/nodebb/src/start.js
- /home/nodebb/forum/nodebb/require-main.js
- /home/nodebb/forum/nodebb/app.js)` is deprecated. Please use `require.main.require("./src/<module_name>")` instead.
    at Object.<anonymous> (/home/nodebb/forum/nodebb/node_modules/nodebb-plugin-audio-embed/library.js:11:24)
2020-04-16T23:29:56.918Z [4568/13903] - info: [plugins/spam-be-gone] Settings loaded
2020-04-16T23:29:56.921Z [4569/13904] - info: [plugins/spam-be-gone] Settings loaded
2020-04-16T23:29:57.029Z [4568/13903] - info: Routes added
2020-04-16T23:29:57.030Z [4567/13902] - info: [plugins/spam-be-gone] Settings loaded
2020-04-16T23:29:57.036Z [4569/13904] - info: Routes added
2020-04-16T23:29:57.043Z [4568/13903] - info: NodeBB Ready
2020-04-16T23:29:57.051Z [4569/13904] - info: NodeBB Ready
2020-04-16T23:29:57.059Z [4568/13903] - info: Enabling 'trust proxy'
2020-04-16T23:29:57.064Z [4568/13903] - info: NodeBB is now listening on: 0.0.0.0:4568
2020-04-16T23:29:57.066Z [4569/13904] - info: Enabling 'trust proxy'
2020-04-16T23:29:57.068Z [4569/13904] - info: NodeBB is now listening on: 0.0.0.0:4569
2020-04-16T23:30:01.733Z [4567/13902] - warn: [plugins] Callback timed out, hook 'static:app.load' in plugin 'nodebb-plugin-emoji'
2020-04-16T23:30:02.233Z [4567/13902] - info: Routes added
2020-04-16T23:30:02.417Z [4567/13902] - info: NodeBB Ready
2020-04-16T23:30:02.420Z [4567/13902] - info: Enabling 'trust proxy'
2020-04-16T23:30:02.426Z [4567/13902] - info: NodeBB is now listening on: 0.0.0.0:4567

[crazycells]

this is the plugin list with order:

nodebb-theme-persona
nodebb-plugin-composer-default
nodebb-plugin-markdown
nodebb-plugin-mentions
nodebb-widget-essentials
nodebb-rewards-essentials
nodebb-plugin-soundpack-default
nodebb-plugin-emoji
nodebb-plugin-dbsearch
nodebb-plugin-btn-spoilers-yesilkart
nodebb-plugin-audio-embed
nodebb-plugin-calendar
nodebb-plugin-canned-responses
nodebb-plugin-emoji-one
nodebb-plugin-extended-markdown
nodebb-plugin-google-analytics
nodebb-plugin-ns-embed
nodebb-plugin-spam-be-gone
nodebb-plugin-osm-map
nodebb-plugin-poll
nodebb-plugin-recent-cards
nodebb-plugin-sso-google
nodebb-plugin-browsing-users
nodebb-plugin-sso-twitter
nodebb-plugin-mega-colors
nodebb-plugin-sso-facebook
nodebb-plugin-custom-pages
nodebb-plugin-tenor-gif
nodebb-plugin-night-mode
nodebb-plugin-2factor
nodebb-plugin-dwnvtr
nodebb-plugin-customize

[sharonyue]

@crazycells said in Error while restarting the forum after several plugin update:

therefore we decided to re-install the forum with an automatic back up from several hours back.

How can you do this automaticly? I am using redis and I have to copy dump.rdb manually.

[crazycells]

@sharonyue said in Error while restarting the forum after several plugin update:

@crazycells said in Error while restarting the forum after several plugin update:

therefore we decided to re-install the forum with an automatic back up from several hours back.

How can you do this automaticly? I am using redis and I have to copy dump.rdb manually.

I think @FreeWorLD should answer this, since he is the main person/coder behind all of our work... As far as I can tell, he wrote some code , so server itself backs up database automatically every 6 hours and deletes oldest ones (we use digital ocean)... As database we use mongodb (I am not sure if it makes difference)...

[gotwf]

@sharonyue I use mongodb and only use redis for sessions. Backup the mongodb via cron'd mongodump.

This guy uses rdiff and a cron job to automate backups of their redis stuff. I've not used this personally.

[FreeWorLD]

@sharonyue said in Error while restarting the forum after several plugin update:

@crazycells said in Error while restarting the forum after several plugin update:

therefore we decided to re-install the forum with an automatic back up from several hours back.

How can you do this automaticly? I am using redis and I have to copy dump.rdb manually.

As just mentioned, mongodb is backed up and the archive is compressed via cron, then sent to an external volume. The older ones also deleted periodically again via cron. Redis is used for sessions.

I am looking into the option of taking multiple snapshots in a day via digitalocean API's to replace this system though. I did this succesfully for the data volumes however I have yet to come up with a way to do the same for the droplet itself as i don't see any digitalocean API for that.

[gotwf]

@crazycells said in Error while restarting the forum after several plugin update:

I think @FreeWorLD should answer this, since he is the main person/coder behind all of our work... As far as I can tell, he wrote some code , so server itself backs up database automatically every 6 hours and deletes oldest ones (we use digital ocean)... As database we use mongodb (I am not sure if it makes difference)...

I explored database options a fair bit. Even looked at PostgreSQL option. From my experimenting I would stick with MongoDB for the main stuff, maybe complimented by an Redis instance for session management (cuz that's what Redis excels at).

[gotwf]

@FreeWorLD said in Error while restarting the forum after several plugin update:

I am looking into the option of taking multiple snapshots in a day via digitalocean API's to replace this system though.

Interesting. I host my stuff on zfs backed filesystems so incremental snapshots are easy. So is mongodump, at least for the size db I am dealing with - hence never bothered with the snaps. Just an image snap every once in a while. Is mongodump taking a long time and killing performance while it does it's thing?

[FreeWorLD]

@gotwf I truly miss zfs