[nodebb-plugin-import] Import your forum to NBB

bentael

@oPignonLibre there is a memory leak, i've known of this issue, but I am refactoring the importer plugin.

here's what you can do if the crash is because you ran out of memory,

go back to NodeBB Admin panel > Plugins > Import
Resume the import by clicking on "Last import was interrupted, try to resume"
the import will start to re-import all the posts, but it will skip the ones already imported

With 1.5M posts, you might have to do this a couple of times - I have definitely imported over 2M posts, and once 60M, it wasn't fun, but doable.

I just moved homes and I am still settling in, I will wrap up my refactor soon, and it will support NodeBB 1.4+ too, sorry the inconvenience.

bentael

wait, sorry I just read that.

Sadly resuming the import creates duplicated threads.

really? I have never had this issue. I will take a look, can you give me a small sample of data, and link me to the mybb exporter that you're using?

bentael

send it to [email protected]

alff0x1f

@bentael yep. I import from phpbb3 last september and remember about duplication.

@oPignonLibre try create big swap before you import

bentael

@alff0x1f Humm.. i will take a look

oPignonLibre

Sorry for the late reply,

I will look at creating some additionnal swap space before importation. I didn't monitor memory usage during initial attempts.

I will try first with a smaller sample and will send it to you if duplication still happens. I used asciicat's mybb exporter.

Thanks

Couby

Hi,

Trying to port a Kunena forum to Nodebb (veaf.org/fr/forums -> community.veaf.org).

I tried https://github.com/eyesofkids/nodebb-plugin-import-kunena .

Looks not too far from the goal .

Users and categories import looks good.
Just a lot of topics and messages missing...

The log (minus repetitive errors for topics and messages import) :
logs.txt

Maybe the problem is with topics import and the related error error:user-too-new ?

Possible to help me solving this issue ? That would consolidate another importer for NodeBB .

Regards.

Arany

Hiho,

is this plugin still maintained? I am asking because we would like to migrate our board with ~25M posts. And the supported version 1.0.3 is nearly two years old. So we would like to do a migration over the course of a month or so, while we keep syncing new/changed content to the new forum, while the old is still in place.

It would not be very practical to move once, and then upgrade, as the forum would have to be down for quite some time, while the migration process is running for a long time. Not even able to change configuration before going live.

Thanks for replying.

Crushed Ice

2018-04-16T18:11:24.857Z [5024] - [31merror[39m: TypeError: path must be a string or Buffer
at Object.fs.readdirSync (fs.js:904:18)
at getSiblingDirectories (C:\Users\root\nodebb\node_modules\nodebb-plugin-import\node_modules\nodebb-plugin-require\index.js:48:19)
at findPackageDirectory (C:\Users\root\nodebb\node_modules\nodebb-plugin-import\node_modules\nodebb-plugin-require\index.js:25:20)
at findNodebbDirectory (C:\Users\root\nodebb\node_modules\nodebb-plugin-import\node_modules\nodebb-plugin-require\index.js:59:9)
at Object.<anonymous> (C:\Users\root\nodebb\node_modules\nodebb-plugin-import\node_modules\nodebb-plugin-require\index.js:62:16)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (C:\Users\root\nodebb\node_modules\nodebb-plugin-import\server\index.js:1:80)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
2018-04-16T18:11:24.857Z [5024] - [33mwarn[39m: [plugins] Unable to parse library for: nodebb-plugin-import

Brophy

Any plans on updating this for the new version of IPB? I've currently got IPB 4.2.6 installed, which I've used for the last 14 years with 1.7 million posts and 32k members.

Would like to migrate the data, if possible.

omega

This plugin supports nBB v1.12.1 according to GitHub

I tried to install this today via ACP and then terminal, but both times I got this funky error. I have no clue what is going on here.

gyp ERR! build error 
gyp ERR! stack Error: not found: make
gyp ERR! stack     at getNotFoundError (/usr/lib/node_modules/npm/node_modules/which/which.js:13:12)
gyp ERR! stack     at F (/usr/lib/node_modules/npm/node_modules/which/which.js:68:19)
gyp ERR! stack     at E (/usr/lib/node_modules/npm/node_modules/which/which.js:80:29)
gyp ERR! stack     at /usr/lib/node_modules/npm/node_modules/which/which.js:89:16
gyp ERR! stack     at /usr/lib/node_modules/npm/node_modules/isexe/index.js:42:5
gyp ERR! stack     at /usr/lib/node_modules/npm/node_modules/isexe/mode.js:8:5
gyp ERR! stack     at FSReqCallback.oncomplete (fs.js:158:21)
gyp ERR! System Linux 4.15.0-47-generic
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/node/nodebb/node_modules/iconv
gyp ERR! node -v v11.15.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 
npm WARN [email protected] requires a peer of textcomplete@^0.14.2 but none is installed. You must install peer dependencies yourself.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/node/.npm/_logs/2019-06-18T13_25_21_688Z-debug.log

The debug log is massive I can post if women want to take a look.

Bri

what OS are you running on?

omega

@Bri ubuntu (18.04 I think.)

<baris>

Try with nodejs 8/10/12

omega

@baris said in [nodebb-plugin-import] Import your forum to NBB:

Try with nodejs 8/10/12

Any point in running npm audit fix on current version of nodejs?

Meanwhile, let's do this - upgrade from node 11.15.0 npm 6.7.0 to node 12.

To note. I experienced (unresolved) erros in past when installing fresh nodejs 12 when it came to the nodeBB install (v1.12).

If that doesn't work I'll try a downgrade to nodejs 10!!!

omega

errrggggghhh.... used n to upgrade/downgrade and with between versions of nodejs , ran the instal via terminal. same deal, now onto

Err msg node12

gyp ERR! build error 
gyp ERR! stack Error: not found: make
gyp ERR! stack     at getNotFoundError (/usr/local/lib/node_modules/npm/node_modules/which/which.js:13:12)
gyp ERR! stack     at F (/usr/local/lib/node_modules/npm/node_modules/which/which.js:68:19)
gyp ERR! stack     at E (/usr/local/lib/node_modules/npm/node_modules/which/which.js:80:29)
gyp ERR! stack     at /usr/local/lib/node_modules/npm/node_modules/which/which.js:89:16
gyp ERR! stack     at /usr/local/lib/node_modules/npm/node_modules/isexe/index.js:42:5
gyp ERR! stack     at /usr/local/lib/node_modules/npm/node_modules/isexe/mode.js:8:5
gyp ERR! stack     at FSReqCallback.oncomplete (fs.js:149:21)
gyp ERR! System Linux 4.15.0-51-generic
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/node/nodebb/node_modules/iconv
gyp ERR! node -v v12.3.1
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 
npm WARN [email protected] requires a peer of textcomplete@^0.14.2 but none is installed. You must install peer dependencies yourself.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/node/.npm/_logs/2019-06-18T20_28_54_773Z-debug.log

Err msg node10

gyp ERR! build error 
gyp ERR! stack Error: not found: make
gyp ERR! stack     at getNotFoundError (/usr/local/lib/node_modules/npm/node_modules/which/which.js:13:12)
gyp ERR! stack     at F (/usr/local/lib/node_modules/npm/node_modules/which/which.js:68:19)
gyp ERR! stack     at E (/usr/local/lib/node_modules/npm/node_modules/which/which.js:80:29)
gyp ERR! stack     at /usr/local/lib/node_modules/npm/node_modules/which/which.js:89:16
gyp ERR! stack     at /usr/local/lib/node_modules/npm/node_modules/isexe/index.js:42:5
gyp ERR! stack     at /usr/local/lib/node_modules/npm/node_modules/isexe/mode.js:8:5
gyp ERR! stack     at FSReqWrap.oncomplete (fs.js:153:21)
gyp ERR! System Linux 4.15.0-51-generic
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/node/nodebb/node_modules/iconv
gyp ERR! node -v v10.16.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 
npm WARN [email protected] requires a peer of textcomplete@^0.14.2 but none is installed. You must install peer dependencies yourself.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/node/.npm/_logs/2019-06-18T20_32_32_662Z-debug.log

Also got a similar error for node8 too.

omega

So far nothing has worked to successfully install this plugin, so I am not sure at this point what to do.

I need to get a discourse base into nodeBB asap, so all help appreciated in advance. Thanks!

omega

To see if maybe, a small maybe, there might be a permissions aspect to this problem I ran this plugin as both sudo & root user.

Sudo

gyp ERR! configure error 
gyp ERR! stack Error: EACCES: permission denied, mkdir '/home/node/nodebb/node_modules/iconv/build'
gyp ERR! System Linux 4.15.0-51-generic
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/node/nodebb/node_modules/iconv
gyp ERR! node -v v11.15.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 
npm WARN [email protected] requires a peer of textcomplete@^0.14.2 but none is installed. You must install peer dependencies yourself.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/node/.npm/_logs/2019-06-19T16_39_04_031Z-debug.log

Root

gyp ERR! build error 
gyp ERR! stack Error: not found: make
gyp ERR! stack     at getNotFoundError (/usr/lib/node_modules/npm/node_modules/which/which.js:13:12)
gyp ERR! stack     at F (/usr/lib/node_modules/npm/node_modules/which/which.js:68:19)
gyp ERR! stack     at E (/usr/lib/node_modules/npm/node_modules/which/which.js:80:29)
gyp ERR! stack     at /usr/lib/node_modules/npm/node_modules/which/which.js:89:16
gyp ERR! stack     at /usr/lib/node_modules/npm/node_modules/isexe/index.js:42:5
gyp ERR! stack     at /usr/lib/node_modules/npm/node_modules/isexe/mode.js:8:5
gyp ERR! stack     at FSReqCallback.oncomplete (fs.js:158:21)
gyp ERR! System Linux 4.15.0-51-generic
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /root/node_modules/iconv
gyp ERR! node -v v11.15.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 
npm WARN enoent ENOENT: no such file or directory, open '/root/package.json'
npm WARN root No description
npm WARN root No repository field.
npm WARN root No README data
npm WARN root No license field.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-06-19T16_39_59_807Z-debug.log

node11

PitaJ

@omega what OS are you on? That's a weird error it seems like make is missing from your system.

Bri

@omega what is the output of this call:

cat /etc/os-release

Do you remember if you installed node or any npm packages as root (which is the user you’re running commands with when you sudo)?

What is your hosting provider?

The errors you’re getting are not typical for node installations so you need to provide more information about your environment.

As an aside I’ve had a lot of trouble with n and will probably migrate back to nvm