How can I migrate from redis to postgresql/mongodb

JaredBusch

Been a year or so since I last posted about this, but now I really need to get off of CentOS 7. I can easily reinstall everything on Fedora 36, but I would like to also get off of redis since the project has changed since I first set this up.

Do I need to give up on NodeBB and move to Discourse or some other backend? I'm not a large site operator, but I've been with NodeBB since version 0.9 or so (November 2015). I like the platform, but I do not like not being on a "normal" design anymore.

Redis was the primary back then, but no longer. I think it would be better if I could convert my system to stay in line with the current install design of NodeBB.

I mean, I an on git v1.19.x (currently 1.19.7) and things still work fine. But for how long will it still work well? Obviously, I need to get off of CentOS 7 because it is causing issues due to old package versions.

gotwf

@JaredBusch Hey Jared, are you not already using PostgreSQL? I am pretty sure tools for migrating from Redis to Mongo?

I would stay away from Discourse. Ping @omega on that. I suspect he will weigh in here with a few "experiences" to share.

@JaredBusch said in How can I migrate from redis to postgresql/mongodb:

do not like not being on a "normal" design anymore...

By "normal", are you referencing PHP and MariaSQL here? Kind of a nightmare security track record there which gives me, at least, pause. I've not used PHP for anything serious in over decade and am told it has gotten "better". Maybe.

If not NodeBB and PHP base app is looking more attractive then I think I'd be kicking the tires on XenForo?

gotwf

@gotwf said in How can I migrate from redis to postgresql/mongodb:

I am pretty sure tools for migrating from Redis to Mongo?

My bad. Seems NodeBB folks offer this for a small fee:

https://community.nodebb.org/topic/10506/migrate-redis-to-mongodb.

Database Migration · Issue #1907 · NodeBB/NodeBB

Is it possible to include a Redis to MongoDB migration option in a future release? Want to back this issue? Place a bounty on it! We accept bounties via Bountysource.

GitHub (github.com)

Although I suspect you'll need to adjust for inflation...

Found this 3rd party script here, fwiw:

GitHub - hyperzlib/nodebb-database-migrate: Migrate data between Redis and MongoDB.

Migrate data between Redis and MongoDB. Contribute to hyperzlib/nodebb-database-migrate development by creating an account on GitHub.

GitHub (github.com)

Good luck!

JaredBusch

@gotwf said in How can I migrate from redis to postgresql/mongodb:

are you not already using PostgreSQL

No, in 2015, the install options were redis or mongo, with redis the actual default. I didn't feel the need to use Mongo as my site would never be large and redis could easily handle it.

@gotwf said in How can I migrate from redis to postgresql/mongodb:

By "normal", are you referencing PHP and MariaSQL here?

By normal, I mean not using the normal NodeBB options that are default today. That means Ubuntu and Mongo according to the docs. https://docs.nodebb.org/installing/os/ubuntu/

JaredBusch

@gotwf said in How can I migrate from redis to postgresql/mongodb:

My bad. Seems NodeBB folks offer this for a small fee:

They have for years. I first asked this question years ago when they switched to mongo as default.

JaredBusch

@gotwf said in How can I migrate from redis to postgresql/mongodb:

Found this 3rd party script here, fwiw:

There are others. None of them have been updated in 2+ years.

gotwf

@JaredBusch Yeah, but I'd thought I remembered seeing a free one somewhere on GitHub. Don't think it was that other one I linked though.

JaredBusch

@gotwf said in How can I migrate from redis to postgresql/mongodb:

Yeah, but I'd thought I remembered seeing a free one

In our conversation from News Year's Day 2021.

https://community.nodebb.org/post/80835

gotwf

@JaredBusch said in How can I migrate from redis to postgresql/mongodb:

@gotwf said in How can I migrate from redis to postgresql/mongodb:

Found this 3rd party script here, fwiw:

There are others. None of them have been updated in 2+ years.

Yeah, seems most folks needing such got to it a bit sooner... Heh...

Still, maybe one of those can be updated and adapted?

JaredBusch

@gotwf said in How can I migrate from redis to postgresql/mongodb:

Still, maybe one of those can be updated and adapted?

Found a pull request for nodebb-plugin-import that gives it postgresql capability.
https://github.com/akhoury/nodebb-plugin-import/pull/240

So I think I will try that an use the "vanilla" import and see what works.

That is a project for the weekend when beer is involved.

gotwf

@JaredBusch Good news for you! Keep us posted, eh?

I am less than keen on mongo these days. At least the corp greed and licensing changes aspect - because if they pull that crap once, they've set precedent, and it is only matter of time before the next round of fsck'ery. Kind of like the old "Screw me once, shame on you. Screw me twice, shame on me." adage. Otherwise it pretty much rocks.

JaredBusch

@JaredBusch said in How can I migrate from redis to postgresql/mongodb:

Found a pull request for nodebb-plugin-import that gives it postgresql capability.

Well, it seem that I cannot install v1.12.1 as required by this plugin.

npm never successfully builds.

[jbusch@obelisk nodebb]$ sudo -u nodebb ./nodebb setup
package.json not found.
Populating package.json...
OK
Dependencies outdated or not yet installed.
Installing them now...

npm notice 
npm notice New minor version of npm available! 8.3.1 -> 8.12.2
npm notice Changelog: https://github.com/npm/cli/releases/tag/v8.12.2
npm notice Run npm install -g [email protected] to update!
npm notice 
npm ERR! code ERESOLVE
npm ERR! ERESOLVE unable to resolve dependency tree
npm ERR! 
npm ERR! While resolving: [email protected]
npm ERR! Found: [email protected]
npm ERR! node_modules/pg
npm ERR!   pg@"^7.4.0" from the root project
npm ERR! 
npm ERR! Could not resolve dependency:
npm ERR! peer pg@"^8" from [email protected]
npm ERR! node_modules/pg-cursor
npm ERR!   pg-cursor@"^2.0.0" from the root project
npm ERR! 
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force, or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
npm ERR! 
npm ERR! See /home/nodebb/.npm/eresolve-report.txt for a full report.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/nodebb/.npm/_logs/2022-06-18T01_31_52_584Z-debug-0.log
Error installing dependencies!
message: Command failed: npm install --production
stdout: null
stderr: null
/opt/nodebb/src/cli/package-install.js:54
		throw e;
		^

Error: Command failed: npm install --production
    at checkExecSyncError (node:child_process:828:11)
    at Object.execSync (node:child_process:902:15)
    at Object.installAll (/opt/nodebb/src/cli/package-install.js:45:9)
    at Object.<anonymous> (/opt/nodebb/src/cli/index.js:60:18)
    at Module._compile (node:internal/modules/cjs/loader:1103:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1155:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18) {
  status: 1,
  signal: null,
  output: [ null, null, null ],
  pid: 1461,
  stdout: null,
  stderr: null
}
[jbusch@obelisk nodebb]$

Built just fine to v2.2.0 though.

[jbusch@obelisk nodebb]$ sudo -u nodebb ./nodebb setup
Dependencies outdated or not yet installed.
Installing them now...

npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: Multer 1.x is affected by CVE-2022-24434. This is fixed in v1.4.4-lts.1 which drops support for versions of Node.js before 6. Please upgrade to at least Node.js 6 and version 1.4.4-lts.1 of Multer. If you need support for older versions of Node.js, we are open to accepting patches that would fix the CVE on the main 1.x release line, whilst maintaining compatibility with Node.js 0.10.
npm WARN deprecated [email protected]: request-promise-native has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: support for ECMAScript is superseded by `uglify-js` as of v3.13.0
npm WARN deprecated [email protected]: You can find the new Popper v2 at @popperjs/core, this package is dedicated to the legacy v1

added 922 packages, and audited 923 packages in 2m

76 packages are looking for funding
  run `npm fund` for details

5 high severity vulnerabilities

Some issues need review, and may require choosing
a different dependency.

Run `npm audit` for details.
OK

2022-06-18T01:40:58.599Z [1600] - info: NodeBB Setup Triggered via Command Line

Welcome to NodeBB v2.2.0!

This looks like a new installation, so you'll have to answer a few questions about your environment before we can proceed.
Press enter to accept the default setting (shown in brackets).