Source forums exporters to NodeBB, I need your DB Dump.

  • Plugin & Theme Dev

    @Steve

    • which exact version of NodeBB are u running against? (don't use master yet, use 0.5.0 or 0.5.0-4 if you have to)
    • also, check the "log on client" option, and paste the last few (5 maybe) logs you see on the browser
    • try running nodebb via either the ./nodebb dev or NODE_ENV=development node app (the latter skips using loader), paste the logs.
    • how big is your DB? is it the one you sent me? I was able to import it in few minutes.

  • Here's the full error log:

    Launching NodeBB in "development" mode.
    To run the production build of NodeBB, please use "forever".
    More Information: https://docs.nodebb.org/en/latest/running/index.html
    info: NodeBB v0.5.0 Copyright (C) 2013-2014 NodeBB Inc.
    info: This program comes with ABSOLUTELY NO WARRANTY.
    info: This is free software, and you are welcome to redistribute it under certain conditions.
    info:
    info: Time: Mon Sep 22 2014 09:42:02 GMT-0400 (EDT)
    info: Initializing NodeBB v0.5.0
    info: * using configuration stored in: /root/mynodebb/config.json
    info: * using redis store at 127.0.0.1:6379
    info: * using themes stored in: /root/mynodebb/node_modules
    info: Base Configuration OK.
    warn: You have no redis password setup!
    morgan deprecated morgan(options): use morgan("default", options) instead src/logger.js:128:22
    morgan deprecated default format: use combined format src/logger.js:128:22
    info: [notifications.init] Registering jobs.
    info: [user.startJobs] Registering User Jobs
    info: Enabling 'trust proxy'
    info: [plugins] Initializing plugins system
    info: [themes] Using theme nodebb-theme-lavender
    info: Middlewares loaded.
    info: [plugins] Found 1 LESS file(s) for plugin nodebb-plugin-markdown
    info: [plugins] Found 1 LESS file(s) for plugin nodebb-plugin-mentions
    info: [plugins] Found 2 js file(s) for plugin nodebb-plugin-mentions
    info: [plugins] Found 1 CSS file(s) for plugin nodebb-widget-essentials
    info: [plugins] Found 3 js file(s) for plugin nodebb-theme-lavender
    info: [plugins] Loaded plugin: nodebb-plugin-markdown
    info: [plugins] Loaded plugin: nodebb-plugin-soundpack-default
    info: [plugins] Loaded plugin: nodebb-plugin-mentions
    info: [plugins] Loaded plugin: nodebb-widget-essentials
    error: ENOENT, no such file or directory '../package.json'
    Error: ENOENT, no such file or directory '../package.json'
    at Object.fs.openSync (fs.js:438:18)
    at Object.fs.readFileSync (fs.js:289:15)
    at Object.me.readFileSync (/root/mynodebb/node_modules/nodebb-plugin-import-vbulletin/node_modules/fs-extra/node_modules/jsonfile/lib/jsonfile.js:31:26)
    at Object.<anonymous> (/root/mynodebb/node_modules/nodebb-plugin-import-vbulletin/admin/index.js:5:11)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at /root/mynodebb/src/plugins.js:153:45
    info: [app] Shutdown (SIGTERM/SIGINT) Initialised.
    info: [app] Database connection closed.
    info: [app] Shutdown complete.
    info: NodeBB v0.5.0 Copyright (C) 2013-2014 NodeBB Inc.
    info: This program comes with ABSOLUTELY NO WARRANTY.
    info: This is free software, and you are welcome to redistribute it under certain conditions.
    info:
    info: Time: Mon Sep 22 2014 09:42:03 GMT-0400 (EDT)
    info: Initializing NodeBB v0.5.0
    info: * using configuration stored in: /root/mynodebb/config.json
    info: * using redis store at 127.0.0.1:6379
    info: * using themes stored in: /root/mynodebb/node_modules
    info: Base Configuration OK.
    warn: You have no redis password setup!
    morgan deprecated morgan(options): use morgan("default", options) instead src/logger.js:128:22
    morgan deprecated default format: use combined format src/logger.js:128:22
    info: [notifications.init] Registering jobs.
    info: [user.startJobs] Registering User Jobs
    info: Enabling 'trust proxy'
    info: [plugins] Initializing plugins system
    info: [themes] Using theme nodebb-theme-lavender
    info: Middlewares loaded.
    info: [plugins] Found 1 LESS file(s) for plugin nodebb-plugin-markdown
    info: [plugins] Found 1 LESS file(s) for plugin nodebb-plugin-mentions
    info: [plugins] Found 2 js file(s) for plugin nodebb-plugin-mentions
    info: [plugins] Found 1 CSS file(s) for plugin nodebb-widget-essentials
    info: [plugins] Found 3 js file(s) for plugin nodebb-theme-lavender
    info: [plugins] Loaded plugin: nodebb-plugin-markdown
    info: [plugins] Loaded plugin: nodebb-plugin-soundpack-default
    info: [plugins] Loaded plugin: nodebb-plugin-mentions
    info: [plugins] Loaded plugin: nodebb-widget-essentials
    error: ENOENT, no such file or directory '../package.json'
    Error: ENOENT, no such file or directory '../package.json'
    at Object.fs.openSync (fs.js:438:18)
    at Object.fs.readFileSync (fs.js:289:15)
    at Object.me.readFileSync (/root/mynodebb/node_modules/nodebb-plugin-import-vbulletin/node_modules/fs-extra/node_modules/jsonfile/lib/jsonfile.js:31:26)
    at Object.<anonymous> (/root/mynodebb/node_modules/nodebb-plugin-import-vbulletin/admin/index.js:5:11)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at /root/mynodebb/src/plugins.js:153:45
    info: [app] Shutdown (SIGTERM/SIGINT) Initialised.
    info: [app] Database connection closed.
    info: [app] Shutdown complete.
    info: NodeBB v0.5.0 Copyright (C) 2013-2014 NodeBB Inc.
    info: This program comes with ABSOLUTELY NO WARRANTY.
    info: This is free software, and you are welcome to redistribute it under certain conditions.
    info:
    info: Time: Mon Sep 22 2014 09:42:04 GMT-0400 (EDT)
    info: Initializing NodeBB v0.5.0
    info: * using configuration stored in: /root/mynodebb/config.json
    info: * using redis store at 127.0.0.1:6379
    info: * using themes stored in: /root/mynodebb/node_modules
    info: Base Configuration OK.
    warn: You have no redis password setup!
    morgan deprecated morgan(options): use morgan("default", options) instead src/logger.js:128:22
    morgan deprecated default format: use combined format src/logger.js:128:22
    info: [notifications.init] Registering jobs.
    info: [user.startJobs] Registering User Jobs
    info: Enabling 'trust proxy'
    info: [plugins] Initializing plugins system
    info: [themes] Using theme nodebb-theme-lavender
    info: Middlewares loaded.
    info: [plugins] Found 1 LESS file(s) for plugin nodebb-plugin-markdown
    info: [plugins] Found 1 LESS file(s) for plugin nodebb-plugin-mentions
    info: [plugins] Found 2 js file(s) for plugin nodebb-plugin-mentions
    info: [plugins] Found 1 CSS file(s) for plugin nodebb-widget-essentials
    info: [plugins] Found 3 js file(s) for plugin nodebb-theme-lavender
    info: [plugins] Loaded plugin: nodebb-plugin-markdown
    info: [plugins] Loaded plugin: nodebb-plugin-soundpack-default
    info: [plugins] Loaded plugin: nodebb-plugin-mentions
    info: [plugins] Loaded plugin: nodebb-widget-essentials
    error: ENOENT, no such file or directory '../package.json'
    Error: ENOENT, no such file or directory '../package.json'
    at Object.fs.openSync (fs.js:438:18)
    at Object.fs.readFileSync (fs.js:289:15)
    at Object.me.readFileSync (/root/mynodebb/node_modules/nodebb-plugin-import-vbulletin/node_modules/fs-extra/node_modules/jsonfile/lib/jsonfile.js:31:26)
    at Object.<anonymous> (/root/mynodebb/node_modules/nodebb-plugin-import-vbulletin/admin/index.js:5:11)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at /root/mynodebb/src/plugins.js:153:45
    info: [app] Shutdown (SIGTERM/SIGINT) Initialised.
    info: [app] Database connection closed.
    info: [app] Shutdown complete.
    info: NodeBB v0.5.0 Copyright (C) 2013-2014 NodeBB Inc.
    info: This program comes with ABSOLUTELY NO WARRANTY.
    info: This is free software, and you are welcome to redistribute it under certain conditions.
    info:
    info: Time: Mon Sep 22 2014 09:42:05 GMT-0400 (EDT)
    info: Initializing NodeBB v0.5.0
    info: * using configuration stored in: /root/mynodebb/config.json
    info: * using redis store at 127.0.0.1:6379
    info: * using themes stored in: /root/mynodebb/node_modules
    info: Base Configuration OK.
    warn: You have no redis password setup!
    morgan deprecated morgan(options): use morgan("default", options) instead src/logger.js:128:22
    morgan deprecated default format: use combined format src/logger.js:128:22
    info: [notifications.init] Registering jobs.
    info: [user.startJobs] Registering User Jobs
    info: Enabling 'trust proxy'
    info: [plugins] Initializing plugins system
    info: [themes] Using theme nodebb-theme-lavender
    info: Middlewares loaded.
    info: [plugins] Found 1 LESS file(s) for plugin nodebb-plugin-markdown
    info: [plugins] Found 1 LESS file(s) for plugin nodebb-plugin-mentions
    info: [plugins] Found 2 js file(s) for plugin nodebb-plugin-mentions
    info: [plugins] Found 1 CSS file(s) for plugin nodebb-widget-essentials
    info: [plugins] Found 3 js file(s) for plugin nodebb-theme-lavender
    info: [plugins] Loaded plugin: nodebb-plugin-markdown
    info: [plugins] Loaded plugin: nodebb-plugin-soundpack-default
    info: [plugins] Loaded plugin: nodebb-plugin-mentions
    info: [plugins] Loaded plugin: nodebb-widget-essentials
    error: ENOENT, no such file or directory '../package.json'
    Error: ENOENT, no such file or directory '../package.json'
    at Object.fs.openSync (fs.js:438:18)
    at Object.fs.readFileSync (fs.js:289:15)
    at Object.me.readFileSync (/root/mynodebb/node_modules/nodebb-plugin-import-vbulletin/node_modules/fs-extra/node_modules/jsonfile/lib/jsonfile.js:31:26)
    at Object.<anonymous> (/root/mynodebb/node_modules/nodebb-plugin-import-vbulletin/admin/index.js:5:11)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at /root/mynodebb/src/plugins.js:153:45
    info: [app] Shutdown (SIGTERM/SIGINT) Initialised.
    info: [app] Database connection closed.
    info: [app] Shutdown complete.

    [loader] Experienced three start attempts in 10 seconds, most likely an error on startup. Halting.


  • @bentael said:

    @Steve

    • which exact version of NodeBB are u running against? (don't use master yet, use 0.5.0 or 0.5.0-4 if you have to)
    • also, check the "log on client" option, and paste the last few (5 maybe) logs you see on the browser
    • try running nodebb via either the ./nodebb dev or NODE_ENV=development node app (the latter skips using loader), paste the logs.
    • how big is your DB? is it the one you sent me? I was able to import it in few minutes.

    0.5.0 version and it's the same DB I sent you plus whatever content they created the past few days so a very small change in size.

    I can't check any options because I can't get into the site at all. I ran the parser after initial connect to database with plugin and got to this point.

  • Plugin & Theme Dev

    @Steve

    I'll test tonight, but you should be able to turn the site back on, NODE_ENV=development node app and see the error.

    if the ssh logins are still the same, I'll check out whats going on your end as well.


  • @bentael said:

    @Steve

    I'll test tonight, but you should be able to turn the site back on, NODE_ENV=development node app and see the error.

    if the ssh logins are still the same, I'll check out whats going on your end as well.

    ssh still the same, and no it still won't go on even with the error 🙂

    I have a feeling it never connected properly to the importer somewhere along the line. It's easy to just wipe and restart but I rather leave it like this to find a solution to keep on record in case it happens to anyone else.

  • Plugin & Theme Dev

    @Steve, so couple of things:

    • You re-installed nodebb-plugin-import-vbulletin instead of nodebb-plugin-import, that's why it was crashing nodebb on startup, but I pushed a fix to keep it from crashing, but that doesn't solve your problem, u need to install the latter plugin.
    • I've had a couple of problems importing your DB to nodebb@0.5.0, but after a couple of tryouts it worked (on my home machine, because I could not figure out your admin passwd); I've hit a couple of Bus error: 10 and a couple of Segmentation fault: 11, looks like that was happening when purging categories from a previous import. Also I hit few TypeErrors in NodeBB core, I am going to investigate these a little more and submit a Pull Request to the NodeBB core source. track issue here issue#51
    • I also found it necessary to change some redis configurations during the import, see this note

    Bottom line, it doesn't look it's 100% ready for large databases yet, sorry about that. If you want to migrate soon, I can personally help you with that, but you need to set a migration date and time, so you can freeze your current site and i can get the migration done meanwhile. It should not take me more than 15-30 minutes (if you're using redis and you have a ready server for that) - talk to me offline.


  • @julian said:

    It could be a server-backed feature, however, just at the current time, it is not.

    Hmm~ that's too bad. For more active forums, it is crucial for users to be able to see which threads they read and which ones they didn't, keeping up with activity just doesn't scale otherwise. Can you say more about the status of this feature? Has it never been considered? Or just postponed to later, or is it EWONTFIX even? This is a showstopper level missing feature for me. =\

    @Bentael Did you ever find a fix for that segfault issue you mentioned?

  • Plugin & Theme Dev

    @Valodim the seg fault issue occurs when scaling up the import process, there were a lot of improvement made the NodeBB master branch, but the import plugin is not compatible yet with it. I will have make some changes and test again, and in a worst case scenario, fork nodebb, and maintain a branch stripped from all uncessary calls that happen during an import.

    track the issue here: https://github.com/akhoury/nodebb-plugin-import/issues/51

    Maybe I'll get around to do it this weekend.


  • @bentael said:

    @Steve, so couple of things:

    • You re-installed nodebb-plugin-import-vbulletin instead of nodebb-plugin-import, that's why it was crashing nodebb on startup, but I pushed a fix to keep it from crashing, but that doesn't solve your problem, u need to install the latter plugin.
    • I've had a couple of problems importing your DB to nodebb@0.5.0, but after a couple of tryouts it worked (on my home machine, because I could not figure out your admin passwd); I've hit a couple of Bus error: 10 and a couple of Segmentation fault: 11, looks like that was happening when purging categories from a previous import. Also I hit few TypeErrors in NodeBB core, I am going to investigate these a little more and submit a Pull Request to the NodeBB core source. track issue here issue#51
    • I also found it necessary to change some redis configurations during the import, see this note

    Bottom line, it doesn't look it's 100% ready for large databases yet, sorry about that. If you want to migrate soon, I can personally help you with that, but you need to set a migration date and time, so you can freeze your current site and i can get the migration done meanwhile. It should not take me more than 15-30 minutes (if you're using redis and you have a ready server for that) - talk to me offline.

    I'm in no rush at all. I'll kill the digital ocean node for now and give it another shot once the issues are worked out. I'm just happy to break things for the sake of others lol. You'll never find me on here bitching and being pushy I just want other vB admins to be able to switch over down the road.

    If you want me to keep it up for testing or anything just let me know.

  • Plugin & Theme Dev

    @Steve that's cool, you can shut off the DigitalOcean node, I have a few running and I test on my home machine.

    So tonight I was able to import your DB from 2 tries only, time took about 10min total.

    but I had to:

    Anyways, I think it's better to wait for 'hashtalk' to merge into master then 0.5.1, i'll keep on testing meanwhile.
    Thanks for being patient.

    looks familiar?

    Imgur

  • NodeBB

    Hashtalk has been merged into master fyi.


  • @bentael said:

    @Steve that's cool, you can shut off the DigitalOcean node, I have a few running and I test on my home machine.

    So tonight I was able to import your DB from 2 tries only, time took about 10min total.

    but I had to:

    Anyways, I think it's better to wait for 'hashtalk' to merge into master then 0.5.1, i'll keep on testing meanwhile.
    Thanks for being patient.

    looks familiar?

    Imgur

    hahah wow nice job!! that's a fairly stock vB database too with the exception of the Garage which we didn't plan on using moving forward since it's more like a graveyard.. people post more recent/daily kind of pics on FB nowadays.

    I'm excited to theme it out offered style. i've already started chopping up pics and whatnot to get it ready to rock.

  • Plugin & Theme Dev

    jeezaloo!

    dat NodeBB master branch is lightning fast!

    @Steve with a combination of NodeBB@master and nodebb-plugin-import@master I was able to import ur DB in less than 5min. Consistently.

    [nodebb-plugin-import] importer.success Importing 632 users.
    [nodebb-plugin-import] importer.success Importing 632/632 users took: 3.63 seconds
    [nodebb-plugin-import] importer.success Importing 59 categories.
    [nodebb-plugin-import] importer.success Importing 59/59 categories took: 1.00 seconds
    [nodebb-plugin-import] importer.success Importing 4586 topics.
    [nodebb-plugin-import] importer.success Importing 4586/4586 topics took: 119.45 seconds
    [nodebb-plugin-import] importer.success Importing 58482 posts.
    [nodebb-plugin-import] importer.success Importing 58482/58482 posts took: 126.70 seconds
    

    big thanks to @baris for some major optimization.

    Oh Also, I added logic to handle mid-import crashes, so you can resume an import now. It is still experimental, but I am targeting the NodeBB 0.5.1 or 0.5.2 release.

  • NodeBB

    Compared to 0.5.0, 0.5.1has some serious optimizations everywhere. I think most of the work went into websockets, mongodb, unread and notification system.

  • Admin

    Damn that's fast. I remember porting a forum with a tenth of those posts and it took like, half an hour or something


  • @bentael said:

    jeezaloo!

    dat NodeBB master branch is lightning fast!

    @Steve with a combination of NodeBB@master and nodebb-plugin-import@master I was able to import ur DB in less than 5min. Consistently.

    [nodebb-plugin-import] importer.success Importing 632 users.
    [nodebb-plugin-import] importer.success Importing 632/632 users took: 3.63 seconds
    [nodebb-plugin-import] importer.success Importing 59 categories.
    [nodebb-plugin-import] importer.success Importing 59/59 categories took: 1.00 seconds
    [nodebb-plugin-import] importer.success Importing 4586 topics.
    [nodebb-plugin-import] importer.success Importing 4586/4586 topics took: 119.45 seconds
    [nodebb-plugin-import] importer.success Importing 58482 posts.
    [nodebb-plugin-import] importer.success Importing 58482/58482 posts took: 126.70 seconds
    

    big thanks to @baris for some major optimization.

    Oh Also, I added logic to handle mid-import crashes, so you can resume an import now. It is still experimental, but I am targeting the NodeBB 0.5.1 or 0.5.2 release.

    looks like i have a late night activity on my hands this evening 🙂

    amazing job you guys are all doing

  • Plugin & Theme Dev

    I would wait for the release, unless you're just playing with it, I am aware of a couple of bugs

    https://github.com/akhoury/nodebb-plugin-import/issues/59
    https://github.com/akhoury/nodebb-plugin-import/issues/58

  • Plugin & Theme Dev

    Memory issues fixed in 0.2.7. Just imported @Valodim 's SMF large DB against NodeBB@0.5.1 release
    But had to disable the Markdown plugin during the import only, because of this

    Screen Shot 2014-10-08 at 12.07.10 AM.png
    Took a good 2 hours on my shitty machine, but that's the price of pagination, at least no server crashing anymore. Start the import and walk away.

    Client side page might crash if you enable client side logging and you get a lot of warnings, but dont care, just refresh the page, the process is still running with or without it, but i the future ill add a feature to recover the logs on the page in case that happens.

    @Steve start from a fresh NodeBB@0.5.1 install, you should be good to go.

  • NodeBB

    The two hours might be a lot less if you were on this or later because each new post is parsing the content as markdown which is unnecessary.

  • Plugin & Theme Dev

    true. I just wanted to release the importer against a stable NodeBB release, even though there was some compromise

Suggested Topics

| |