How can I migrate from redis to postgresql/mongodb
-
Been a year or so since I last posted about this, but now I really need to get off of CentOS 7. I can easily reinstall everything on Fedora 36, but I would like to also get off of redis since the project has changed since I first set this up.
Do I need to give up on NodeBB and move to Discourse or some other backend? I'm not a large site operator, but I've been with NodeBB since version 0.9 or so (November 2015). I like the platform, but I do not like not being on a "normal" design anymore.
Redis was the primary back then, but no longer. I think it would be better if I could convert my system to stay in line with the current install design of NodeBB.
I mean, I an on git v1.19.x (currently 1.19.7) and things still work fine. But for how long will it still work well? Obviously, I need to get off of CentOS 7 because it is causing issues due to old package versions.
-
@JaredBusch Hey Jared, are you not already using PostgreSQL? I am pretty sure tools for migrating from Redis to Mongo?
I would stay away from Discourse. Ping @omega on that. I suspect he will weigh in here with a few "experiences" to share.
@JaredBusch said in How can I migrate from redis to postgresql/mongodb:
do not like not being on a "normal" design anymore...
By "normal", are you referencing PHP and MariaSQL here? Kind of a nightmare security track record there which gives me, at least, pause. I've not used PHP for anything serious in over decade and am told it has gotten "better". Maybe.
If not NodeBB and PHP base app is looking more attractive then I think I'd be kicking the tires on XenForo?
-
@gotwf said in How can I migrate from redis to postgresql/mongodb:
I am pretty sure tools for migrating from Redis to Mongo?
My bad. Seems NodeBB folks offer this for a small fee:
https://community.nodebb.org/topic/10506/migrate-redis-to-mongodb.
https://github.com/NodeBB/NodeBB/issues/1907
Although I suspect you'll need to adjust for inflation...
Found this 3rd party script here, fwiw:
https://github.com/hyperzlib/nodebb-database-migrate
Good luck!
-
@gotwf said in How can I migrate from redis to postgresql/mongodb:
are you not already using PostgreSQL
No, in 2015, the install options were redis or mongo, with redis the actual default. I didn't feel the need to use Mongo as my site would never be large and redis could easily handle it.
@gotwf said in How can I migrate from redis to postgresql/mongodb:
By "normal", are you referencing PHP and MariaSQL here?
By normal, I mean not using the normal NodeBB options that are default today. That means Ubuntu and Mongo according to the docs. https://docs.nodebb.org/installing/os/ubuntu/
-
@gotwf said in How can I migrate from redis to postgresql/mongodb:
My bad. Seems NodeBB folks offer this for a small fee:
They have for years. I first asked this question years ago when they switched to mongo as default.
-
@gotwf said in How can I migrate from redis to postgresql/mongodb:
Found this 3rd party script here, fwiw:
There are others. None of them have been updated in 2+ years.
-
@JaredBusch Yeah, but I'd thought I remembered seeing a free one somewhere on GitHub. Don't think it was that other one I linked though.
-
@gotwf said in How can I migrate from redis to postgresql/mongodb:
Yeah, but I'd thought I remembered seeing a free one
In our conversation from News Year's Day 2021.
-
@JaredBusch said in How can I migrate from redis to postgresql/mongodb:
@gotwf said in How can I migrate from redis to postgresql/mongodb:
Found this 3rd party script here, fwiw:
There are others. None of them have been updated in 2+ years.
Yeah, seems most folks needing such got to it a bit sooner... Heh...
Still, maybe one of those can be updated and adapted?
-
@gotwf said in How can I migrate from redis to postgresql/mongodb:
Still, maybe one of those can be updated and adapted?
Found a pull request for
nodebb-plugin-importthat gives it postgresql capability.
https://github.com/akhoury/nodebb-plugin-import/pull/240So I think I will try that an use the "vanilla" import and see what works.
That is a project for the weekend when beer is involved.
-
@JaredBusch Good news for you! Keep us posted, eh?
I am less than keen on mongo these days. At least the corp greed and licensing changes aspect - because if they pull that crap once, they've set precedent, and it is only matter of time before the next round of fsck'ery. Kind of like the old "Screw me once, shame on you. Screw me twice, shame on me." adage. Otherwise it pretty much rocks.
-
@JaredBusch said in How can I migrate from redis to postgresql/mongodb:
Found a pull request for
nodebb-plugin-importthat gives it postgresql capability.Well, it seem that I cannot install v1.12.1 as required by this plugin.
npmnever successfully builds.[jbusch@obelisk nodebb]$ sudo -u nodebb ./nodebb setup package.json not found. Populating package.json... OK Dependencies outdated or not yet installed. Installing them now... npm notice npm notice New minor version of npm available! 8.3.1 -> 8.12.2 npm notice Changelog: https://github.com/npm/cli/releases/tag/v8.12.2 npm notice Run npm install -g [email protected] to update! npm notice npm ERR! code ERESOLVE npm ERR! ERESOLVE unable to resolve dependency tree npm ERR! npm ERR! While resolving: [email protected] npm ERR! Found: [email protected] npm ERR! node_modules/pg npm ERR! pg@"^7.4.0" from the root project npm ERR! npm ERR! Could not resolve dependency: npm ERR! peer pg@"^8" from [email protected] npm ERR! node_modules/pg-cursor npm ERR! pg-cursor@"^2.0.0" from the root project npm ERR! npm ERR! Fix the upstream dependency conflict, or retry npm ERR! this command with --force, or --legacy-peer-deps npm ERR! to accept an incorrect (and potentially broken) dependency resolution. npm ERR! npm ERR! See /home/nodebb/.npm/eresolve-report.txt for a full report. npm ERR! A complete log of this run can be found in: npm ERR! /home/nodebb/.npm/_logs/2022-06-18T01_31_52_584Z-debug-0.log Error installing dependencies! message: Command failed: npm install --production stdout: null stderr: null /opt/nodebb/src/cli/package-install.js:54 throw e; ^ Error: Command failed: npm install --production at checkExecSyncError (node:child_process:828:11) at Object.execSync (node:child_process:902:15) at Object.installAll (/opt/nodebb/src/cli/package-install.js:45:9) at Object.<anonymous> (/opt/nodebb/src/cli/index.js:60:18) at Module._compile (node:internal/modules/cjs/loader:1103:14) at Object.Module._extensions..js (node:internal/modules/cjs/loader:1155:10) at Module.load (node:internal/modules/cjs/loader:981:32) at Function.Module._load (node:internal/modules/cjs/loader:822:12) at Module.require (node:internal/modules/cjs/loader:1005:19) at require (node:internal/modules/cjs/helpers:102:18) { status: 1, signal: null, output: [ null, null, null ], pid: 1461, stdout: null, stderr: null } [jbusch@obelisk nodebb]$Built just fine to v2.2.0 though.
[jbusch@obelisk nodebb]$ sudo -u nodebb ./nodebb setup Dependencies outdated or not yet installed. Installing them now... npm WARN deprecated [email protected]: this library is no longer supported npm WARN deprecated [email protected]: Multer 1.x is affected by CVE-2022-24434. This is fixed in v1.4.4-lts.1 which drops support for versions of Node.js before 6. Please upgrade to at least Node.js 6 and version 1.4.4-lts.1 of Multer. If you need support for older versions of Node.js, we are open to accepting patches that would fix the CVE on the main 1.x release line, whilst maintaining compatibility with Node.js 0.10. npm WARN deprecated [email protected]: request-promise-native has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142 npm WARN deprecated [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details. npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142 npm WARN deprecated [email protected]: support for ECMAScript is superseded by `uglify-js` as of v3.13.0 npm WARN deprecated [email protected]: You can find the new Popper v2 at @popperjs/core, this package is dedicated to the legacy v1 added 922 packages, and audited 923 packages in 2m 76 packages are looking for funding run `npm fund` for details 5 high severity vulnerabilities Some issues need review, and may require choosing a different dependency. Run `npm audit` for details. OK 2022-06-18T01:40:58.599Z [1600] - info: NodeBB Setup Triggered via Command Line Welcome to NodeBB v2.2.0! This looks like a new installation, so you'll have to answer a few questions about your environment before we can proceed. Press enter to accept the default setting (shown in brackets).
-
G gotwf referenced this topic on