Failed migration of NodeBB

finid

I have an outdated installation of NodeBB that I decided to migrate to a new server. I thought it would be much easier to install a fresh copy of NodeBB on the new server, then copy over the Uploads folder and the database and be done with it, just as with WordPress.

However, when I did that, started NodeBB and tried to access, I got an Nginx 502 Bad Gateway error. Later determined that NodeBB failed to start.

In the error log are these entries:

Clustering enabled: Spinning up 1 process(es).

20/7 23:59 [30485] - info: Time: Mon Jul 20 2015 23:59:35 GMT-0400 (EDT)
20/7 23:59 [30485] - info: Initializing NodeBB v0.7.0
20/7 23:59 [30485] - warn: Your NodeBB schema is out-of-date. Please run the following command to bring your dataset up to spec:
20/7 23:59 [30485] - warn:     ./nodebb upgrade
[cluster] Child Process (30485) has exited (code: 0, signal: null)

So I ran the upgrade command, but that also failed with:

21/7 00:07 [30780] - info: Beginning database schema update
21/7 00:07 [30780] - error: [upgrade] NodeBB upgrade could not complete, as your database schema is too far out of date.
21/7 00:07 [30780] - error: [upgrade]   Please ensure that you did not skip any minor version upgrades.
21/7 00:07 [30780] - error: [upgrade]   (e.g. v0.1.x directly to v0.3.x)

The imported database was from a NodeBB 0.4.x installation and the fresh installation is NodeBB 0.7.x.

How do I update the database (Redis) schema?

Pinging the gurus > @baris, @Julian-H-Lam, @psychobunny

julian

Hiya ~~@planner~~ @finid

# Back up your database first!!!
$ git checkout v0.5.x
$ node app --upgrade   # upgrade the schema only
$ git checkout v0.6.x
$ node app --upgrade   # upgrade the schema only
$ git checkout v0.7.x
$ ./nodebb upgrade     # upgrade dependencies and schema

Now, in between some of those upgrade steps, you might have to run npm install. YMMV.

For simplicity's sake, we don't store upgrade scripts in between minor versions, otherwise src/upgrade.js would be many thousands of lines in length

So you'll have to dial it back a bit, and upgrade progressively through minor versions.

finid

@julian, yeah, @planner died and reincarnated as @finid

I actually started on that step-wise upgrade path, but had to attend to other matters before I could post the result of my efforts.

Upgrading from 0.4.x to 0.5.x completed successfully. However, trying to go from 0.5.x to 0.6.x failed with the error shown in the code block below:

npm ERR! install Couldn't read dependencies
npm ERR! Failed to parse json
npm ERR! Unexpected token <
npm ERR! File: /var/www/sites/forum/package.json
npm ERR! Failed to parse package.json data.
npm ERR! package.json must be actual JSON, not just JavaScript.
npm ERR! 
npm ERR! This is not a bug in npm.
npm ERR! Tell the package author to fix their package.json file. JSON.parse

npm ERR! System Linux 3.13.0-52-generic
npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install"
npm ERR! cwd /var/www/sites/forum
npm ERR! node -v v0.10.25
npm ERR! npm -v 1.3.10
npm ERR! file /var/www/sites/forum/package.json
npm ERR! code EJSONPARSE
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /var/www/sites/forum/npm-debug.log
npm ERR! not ok code 0
npm ERR! Failed to parse json
npm ERR! Unexpected token <
npm ERR! File: /var/www/sites/forum/package.json
npm ERR! Failed to parse package.json data.
npm ERR! package.json must be actual JSON, not just JavaScript.
npm ERR! 
npm ERR! This is not a bug in npm.
npm ERR! Tell the package author to fix their package.json file. JSON.parse

npm ERR! System Linux 3.13.0-52-generic
npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "i" "nodebb-theme-vanilla" "nodebb-theme-lavender" "nodebb-widget-essentials"
npm ERR! cwd /var/www/sites/forum
npm ERR! node -v v0.10.25
npm ERR! npm -v 1.3.10
npm ERR! file /var/www/sites/forum/package.json
npm ERR! code EJSONPARSE
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /var/www/sites/forum/npm-debug.log
npm ERR! not ok code 0

Apparently, package.json got so mangled that it was causing the upgrade to fail.

Heer's what the file looks like:

{
  "name": "nodebb",
  "license": "GPLv3 or later",
  "description": "NodeBB Forum",
<<<<<<< HEAD
  "version": "0.6.1",
=======
  "version": "0.5.7",
>>>>>>> origin/v0.5.x
  "homepage": "http://www.nodebb.org",
  "repository": {
    "type": "git",
    "url": "https://github.com/NodeBB/NodeBB/"
  },
  "main": "app.js",
  "scripts": {
    "start": "./nodebb start",
    "stop": "./nodebb stop",
    "test": "mocha ./tests -t 10000"
  },
  "dependencies": {
    "async": "~0.9.0",
    "bcryptjs": "~2.1.0",
    "body-parser": "^1.9.0",
    "compression": "^1.1.0",
    "connect-ensure-login": "^0.1.1",
    "connect-flash": "^0.1.1",
    "connect-multiparty": "^1.2.4",
    "cookie-parser": "^1.3.3",
    "cron": "^1.0.5",
    "csurf": "^1.6.1",
    "daemon": "~1.1.0",
    "express": "^4.9.5",
    "express-session": "^1.8.2",
    "gm": "1.17.0",
    "gravatar": "^1.1.0",
    "heapdump": "^0.3.0",
    "less": "^2.0.0",
    "logrotate-stream": "^0.2.3",
    "mkdirp": "~0.5.0",
    "morgan": "^1.3.2",
    "nconf": "~0.7.1",
    "nodebb-plugin-dbsearch": "^0.1.0",
    "nodebb-plugin-markdown": "^0.8.0",
    "nodebb-plugin-mentions": "^0.9.0",
    "nodebb-plugin-soundpack-default": "~0.1.1",
<<<<<<< HEAD
    "nodebb-plugin-spam-be-gone": "^0.4.0",
    "nodebb-theme-lavender": "^0.2.0",
    "nodebb-theme-vanilla": "^0.2.0",
    "nodebb-widget-essentials": "~0.2.0",
=======
    "nodebb-theme-lavender": "0.1.13",
    "nodebb-theme-vanilla": "0.1.30",
    "nodebb-widget-essentials": "~0.1.1",
>>>>>>> origin/v0.5.x
    "npm": "^2.1.4",
    "passport": "^0.2.1",
    "passport-local": "1.0.0",
    "prompt": "^0.2.14",
    "request": "^2.44.0",
    "rimraf": "~2.2.6",
    "rss": "^1.0.0",
    "semver": "^4.0.3",
    "serve-favicon": "^2.1.5",
    "sitemap": "^0.7.4",
    "socket.io": "^1.2.1",
    "socket.io-client": "^1.2.1",
    "socket.io-redis": "^0.1.3",
    "socketio-wildcard": "~0.1.1",
    "string": "^3.0.0",
    "templates.js": "^0.1.12",
    "uglify-js": "git+https://github.com/julianlam/UglifyJS2.git",
    "underscore": "~1.7.0",
    "validator": "~3.28.0",
    "winston": "^0.9.0",
    "xregexp": "~2.0.0"
  },
  "devDependencies": {
    "mocha": "~1.13.0"
  },
  "bugs": {
    "url": "https://github.com/NodeBB/NodeBB/issues"
  },
  "engines": {
    "node": ">=0.10"
  },
  "maintainers": [
    {
      "name": "Andrew Rodrigues",
      "email": "[email protected]",
      "url": "https://github.com/psychobunny"
    },
    {
      "name": "Julian Lam",
      "email": "[email protected]",
      "url": "https://github.com/julianlam"
    },
    {
      "name": "Barış Soner Uşaklı",
      "email": "[email protected]",
      "url": "https://github.com/barisusakli"
    }
  ]
}

Tried to use http://jsonlint.com, but there were key-value pairs that I could not get right.

Is there a version of package.json I could pull or could you take a stab at helping validate it?

julian

~~@planner~~ @finid Sure. Now that your schema is roughly ~v0.5.x:

$ git checkout v0.6.x
$ git reset --hard origin/v0.6.x
$ node app --upgrade

finid

@julian, looks like I replied to myself instead to you

Shows how long I've used this platform...

finid

@julian, I told you, man! ~@planner died of unknown causes. But he did leave behind a boat load of bitcoins in a Swiss account. So I'm shopping for a pacific island right now

To this particular matter, I'll continue where I left off later tonigh, because the server is laready life and the upgrade uses up lots of RAM.

finid

@julian, So upgrade to 0.6 completed successfully.

However, upgrade to 0.7 almost completed. At the tail end, it failed with this message:

21/7 23:40 [30598] - info: [2015/2/23] Activating NodeBB Essential Rewards
21/7 23:40 [30598] - info: [2015/02/24] Upgrading plugins:active to sorted set
21/7 23:40 [30598] - info: [2015/02/24] Upgrading plugins:active to sorted set done
21/7 23:40 [30598] - info: [2015/02/24] Upgrading privilege groups to system groups
21/7 23:40 [30598] - info: [2015/02/24] Upgrading privilege groups to system groups done
21/7 23:40 [30598] - info: [2015/02/25] Upgrading menu items to dynamic navigation system
21/7 23:40 [30598] - info: [2015/02/25] Upgrading menu items to dynamic navigation system done
21/7 23:40 [30598] - info: [2015/05/07] Upgrading uid mappings to sorted set
21/7 23:40 [30598] - info: [2015/05/07] Upgrading uid mappings to sorted set done
21/7 23:40 [30598] - info: [2015/05/08] Fixing emails
21/7 23:41 [30598] - error: undefined

Undefined error. Perhaps the contents of the log could shed some light:

at Array.forEach (native)
    at Function.use (/var/forum/node_modules/express/lib/router/index.js:436:13)
    at EventEmitter.<anonymous> (/var/forum/node_modules/express/lib/application.js:187:21)
    at Array.forEach (native)
    at EventEmitter.use (/var/forum/node_modules/express/lib/application.js:184:7)
    at module.exports (/var/forum/src/routes/index.js:144:6)
    at require.exec.cwd (/var/forum/src/webserver.js:63:3)
[cluster] Child Process (9241) has disconnected
[cluster] Child Process (9241) has exited (code: 8, signal: null)
[cluster] Spinning up another process...

Any clues?

finid

Ok, so I decide to skip 0.7 and use the last known good upgrade (0.6) until I get a response on the blocking issue.

However, when I stat NodeBB, it starts, but I get a 502 Bad Gateway error when I try to access the site. From the log, I get these:

Clustering enabled: Spinning up 1 process(es).

22/7 02:00 [2356] - info: Time: Wed Jul 22 2015 02:00:28 GMT-0400 (EDT)
22/7 02:00 [2356] - info: Initializing NodeBB v0.6.1
22/7 02:00 [2356] - error: [plugins] NodeBB encountered a problem while loading plugins WRONGTYPE Operation against a key holding the wrong kind of value

Any clues?

finid

@finid, So I made the decision to disable all the plugins, now knowing which is causing the problem.

Now I can NodeBB starts fine, I can access the site, but it's a blank site. No posts or users at all. Check the uploads folders that all is intact. Also verified that the database files (.rdb and .aof) are still in place and the same size as the original site.

Any ideas @julian

Bri

@finid I do believe someone else was having trouble with upgrading to 0.7.x from 0.6.x. It wasn't clear which minor version of 0.6.x you switched to when you were upgrading, so this might help you: It was discovered that you'd have to upgrade from 0.6.0 to 0.6.1 so the db schema could complete correctly. @julian can probably explain why if you need details, but @WildPointer had success going through these steps in this post: https://community.nodebb.org/topic/4700/solved-upgrading-nodebb-0-6-0-to-0-7-0/20

HTH

julian

@finid said:

Now I can NodeBB starts fine, I can access the site, but it's a blank site. No posts or users at all. Check the uploads folders that all is intact. Also verified that the database files (.rdb and .aof) are still in place and the same size as the original site.

Ensure that the config.json still points to the same database... if it's a blank site, then NodeBB is reading an empty database...

finid

@BDHarrington7, I'm on v0.6.1-15, but I think I'll skip upgrading to 0.7.x until I can get the whole site back up.

finid

@julian said:

Ensure that the config.json still points to the same database... if it's a blank site, then NodeBB is reading an empty database...

Before upgrading, config.json was pointing to Redis DB 0. After upgrading, it's still pointing to Redis DB 0. The configured Redis password still matches.

So not sure, how to proceed from here.