Migrating to NodeBB - good stuff

[bentael]

Sorry a long ass reply with snapshots, it's 5:30am don't judge me.

@medwards

I think nodebb-plugin-import is ready for your set of eyes looking at it, if you can test it out with your real data (didn't publish to npm yet, still running some tests before I do that)

here's what I did:

• Basically made it compatible with 0.3.x (latest commit was 46e29dfb2b841ad4919059886b8c68f1c21da77e), however, and unfortunately, you still need to comment-out/change few lines of code in NodeBB's src, please read this readme section very carefully and click on each image to view the lines numbers to comment. Keep in mind that these line numbers are based on the commit-snapshot i mentioned in the beginning of this long sentence.
• added both html-to-md and bbcode-to-md support to config.convert so you can choose either, and the performance is not that bad 8minutes (mac mini, i5, 16gb) to import that sample dump you sent me, look at the config section in the readme.
• fixed locked issue, now ALL the topics are assumed unlocked with a flag, but then I relock them after all the posts get imported in
• fixed pinned/unpinned issue
• guarded against some the potential Errors you mentioned.

give it a try, let me know, don't forget to revert the NodeBB src changes right after.
Thanks for sending these sample dump files btw, nice work on the automated generation. I couldn't test the bbcode conversion since the dump content were all clear text, but I stole a function from here, then modified a little, see code here, I'll let you test it out.

Also, I am not sure what's the difference between bb Code and vB code, can anyone explain?

#Snapshots:

locking works with a reply posted to it

pinned threads now float to the top

general migrated data looked good

[julian]

@bentael Thanks so much for the work you're putting into this. You are providing an invaluable service to the community

[bentael]

@julian de nada.
hey do you the difference between vB Code and bb code ?

[psychobunny]

wow, awesome @bentael

vB code
http://www.vbulletin.org/forum/misc.php?do=bbcode

vs

ubb code
http://www.ubbwiki.com/index.php/UBBCode

imo the one you used for ubb should be more than satisfactory for vB sites (just need to change the : to =)

[bentael]

@psychobunny actually your second link should be http://en.wikipedia.org/wiki/BBCode

UBB used straight HTML, at least that's what mine did.

Anyways, looking at the bbcode vs the vB code, they look the same, to me al least. so the bbcode-to-html conversion should work fine.

[medwards]

@bentael Followed your instructions including code modifications - worked great! Pinned and locked threads are coming through as expected now, and the bbcode conversion sort of worked.

The tags that are included in that regex list were properly replaced - I think I'm going to have to look through that link @psychobunny provided to ubbcode and add a few extra tags: [quote], [center] and maybe a few others, but some of the formatting came across nicely.

Thanks for all the great work! I'll circle back once I get a chance to shore up the code converter to provide my updates. Your work got me to a nice stopping point for now and I'm going to have to prioritize our authentication/user creation stuff so I'll hopefully get back to the bbcode conversion a bit later in the week.

[bentael]

@medwards great to hear ! feel free to edit that RegExp, I accept pull requests - and if it's a new format, we can add that too.

Don't forget to check with your superiors to open-source that "vB exporter" when you have a chance. (name it something like nodebb-plugin-import-vbulletin to keep the convention) - if you don't have the time to write a full read-me and docs, that's fine, I can help out on that.

[medwards]

@bentael quick update: first draft of the vbexporter has been submitted for open-sourcing approval.

It needs a bit more cleanup for general consumption ( it's my first node_module "release" ) but it's a decent starting point.

Also I'm going to go ahead and fork your importer. Once I've got anything worth contributing back I'll figure out how to issue a pull request ( first time for that, too ).

Thanks for all your work (and the node team too)! I'm excited to be able to contribute something back.

[bentael]

@medwards sweet! once it's opened, I will fork it and give you a hand polishing it up.

[medwards]

@bentael @psychobunny @julian

So I have approval to open source the vb exporter. It's rough at the moment and I won't have time to improve it a for a few weeks, but at least it'll be out there. I'll post a link later today.

Quick question, also. After the most recent run of the importer my "admin" user (uid:1) does not appear to be an admin user. He's in the gid:2:members set which is the administrators group. I found an old post that said to add the id to the "administrators" global key in redis, but that doesn't appear to exist anymore.

Any pointers?

[bentael]

@medwards i may have an idea why, investigating..

[bentael]

/facepalm
https://github.com/akhoury/nodebb-plugin-import/blob/master/lib/import.js#L522 that's a left over from my early nib code.

but that may require a little more work to fix that i thought. If you already migrated, try to make the use an Admin manually.
go to the /admin/users/search route, find that user, click on Make Admin.

[bentael]

oh wait, im dumb, if do not have any admin users, you can't go the to /admin page.

@medwards it's 'group:gid' global object,

redis-cli hgetall group:gid
# you should get an 'adminisitrator' field with value, most likely 2

# then, assuming gid is 2, and the user.id that you want to make admin is also 2 (just a coincidence), so really the command is sadd gid:{gid}:members {uid}
sadd gid:2:members 2

[medwards]

@bentael I think the database is already in the state you are trying to get me to put it in:

redis> hgetall group:gid

1. "registered-users"
2. "1"
3. "administrators"
4. "2"

redis> smembers gid:2:members

1. "1"

redis> hget user:1 username
"admin"

redis>

The user "admin" (uid:1) is already in the gid:2:members set (which is the "administrators" group).

Are we sure the latest version didn't change the way privileges are allocated?
@psychobunny

[bentael]

@medwards your data looks correct, so you have 1 admin only, If you use the --flush flag with import, it WILL create the first admin user as admin:password because it is required by the NodeBB setup, basically just calling

node app --setup="{ escaped JSON goes here}"

to make sure you have the right DB schema, here are the default values. https://github.com/akhoury/nodebb-plugin-import/blob/master/lib/import.js#L79

###tl;dr
no matter if you have admins or in the imported DATA, the import WILL create the first admin for you, and tries to make YOUR imported-admins admins,

[bentael]

that's why your first user with id 1 has id 2, so .. all ids are incremented by one.
One way to avoid this is to set your main admin as the default Admin in the config.setupVal values, and delete its _uid.1 file. this way, it will be the one to be imported as the main admin, and all of its posts will be owned by that same one.

[medwards]

@bentael Sorry for the delayed reply.

I understand everything you said, but I don't think that is my problem. I am using the "admin" user that is uid:1.

After digging into the code I was able to figure this out. It looks like the group that grants admin access has changed from "administrators" to "Administrators" which is case sensitive. I was able to solve my problem with

hset group:gid Administrators 2

I based this on line 752 in user.js

groups.getGidFromName('Administrators', function(err, gid) {

which sort of just creates another name for the same group... I think. Not sure if this would work throughout the codebase or not.

[julian]

Whoa, actually it's the reverse. "Administrators" changed to "administrators", so if your build is expecting admins to be in group "Administrators", you should probably grab the latest codebase and try his importer again

[medwards]

I see what you mean in the commit log. Based on this:

@julian said:

@bentael @medwards For the most part, we're limiting refactors to minor version bumps, and as the next version is 0.3.1, you should be fine if @bentael fixes up the importer for edge, even if 0.3.1 isn't released by then.

I was running the data import using the version he specified, but then launching the web app with v0.3.0 expecting them to be data compatible. I'll have to upgrade the code I'm using to launch the web server - my mistake.

[bentael]

@medwards yea, you will need to use the version I specified for both import and run, but then after you can just ./nodebb upgrade to be on the latest.
as soon as 0.3.1 is out, I will stablize the importer again, and with every minor release for here on (0.4.0, 0.5.0 etc..)