Migrating to NodeBB - good stuff

[medwards]

@bentael We are not live.

I'm debugging the data import process so when it comes time to cut over I can shut down the old forum, do a data dump -> import and update DNS stuff etc. to cut-over.

I'll try to get you some example data before this weekend if that's your timeline to do some patching. I'll need to hand edit to remove content and just leave the skeleton, but happy to do it so you can provide input.

[bentael]

ok, email me at [email protected]
also, if you find a good NodeJS "VB Code to Markdown" I will add a config to choose which format to convert from.

but just to be clear after I patch it, the import plugin will NOT support 0.3.0 and lower, only 0.3.x-edge. Are you willing to wait for 0.3.1 to import again?

[medwards]

@bentael I've heard discussions that we'd like to go live in the next 1-2 weeks and will want some time to confirm I have a stable data migration pipeline.

Not sure what the timeline on 0.3.1 is...?

I guess I should be okay to run edge for now until it get's there, though...

[bentael]

@medwards oh - ok. Looks like 0.3.1 is not due till like a month.

tell you what, I'll patch up to 0.3.0 so you can import to that, then it's just a matter of ./nodebb upgrade from there.

[julian]

@bentael @medwards For the most part, we're limiting refactors to minor version bumps, and as the next version is 0.3.1, you should be fine if @bentael fixes up the importer for edge, even if 0.3.1 isn't released by then.

[bentael]

@julian got you, i actually prefer doing that than falling back to 0.3.0

[bentael]

Sorry a long ass reply with snapshots, it's 5:30am don't judge me.

@medwards

I think nodebb-plugin-import is ready for your set of eyes looking at it, if you can test it out with your real data (didn't publish to npm yet, still running some tests before I do that)

here's what I did:

• Basically made it compatible with 0.3.x (latest commit was 46e29dfb2b841ad4919059886b8c68f1c21da77e), however, and unfortunately, you still need to comment-out/change few lines of code in NodeBB's src, please read this readme section very carefully and click on each image to view the lines numbers to comment. Keep in mind that these line numbers are based on the commit-snapshot i mentioned in the beginning of this long sentence.
• added both html-to-md and bbcode-to-md support to config.convert so you can choose either, and the performance is not that bad 8minutes (mac mini, i5, 16gb) to import that sample dump you sent me, look at the config section in the readme.
• fixed locked issue, now ALL the topics are assumed unlocked with a flag, but then I relock them after all the posts get imported in
• fixed pinned/unpinned issue
• guarded against some the potential Errors you mentioned.

give it a try, let me know, don't forget to revert the NodeBB src changes right after.
Thanks for sending these sample dump files btw, nice work on the automated generation. I couldn't test the bbcode conversion since the dump content were all clear text, but I stole a function from here, then modified a little, see code here, I'll let you test it out.

Also, I am not sure what's the difference between bb Code and vB code, can anyone explain?

#Snapshots:

locking works with a reply posted to it

pinned threads now float to the top

general migrated data looked good

[julian]

@bentael Thanks so much for the work you're putting into this. You are providing an invaluable service to the community

[bentael]

@julian de nada.
hey do you the difference between vB Code and bb code ?

[psychobunny]

wow, awesome @bentael

vB code
http://www.vbulletin.org/forum/misc.php?do=bbcode

vs

ubb code
http://www.ubbwiki.com/index.php/UBBCode

imo the one you used for ubb should be more than satisfactory for vB sites (just need to change the : to =)

[bentael]

@psychobunny actually your second link should be http://en.wikipedia.org/wiki/BBCode

UBB used straight HTML, at least that's what mine did.

Anyways, looking at the bbcode vs the vB code, they look the same, to me al least. so the bbcode-to-html conversion should work fine.

[medwards]

@bentael Followed your instructions including code modifications - worked great! Pinned and locked threads are coming through as expected now, and the bbcode conversion sort of worked.

The tags that are included in that regex list were properly replaced - I think I'm going to have to look through that link @psychobunny provided to ubbcode and add a few extra tags: [quote], [center] and maybe a few others, but some of the formatting came across nicely.

Thanks for all the great work! I'll circle back once I get a chance to shore up the code converter to provide my updates. Your work got me to a nice stopping point for now and I'm going to have to prioritize our authentication/user creation stuff so I'll hopefully get back to the bbcode conversion a bit later in the week.

[bentael]

@medwards great to hear ! feel free to edit that RegExp, I accept pull requests - and if it's a new format, we can add that too.

Don't forget to check with your superiors to open-source that "vB exporter" when you have a chance. (name it something like nodebb-plugin-import-vbulletin to keep the convention) - if you don't have the time to write a full read-me and docs, that's fine, I can help out on that.

[medwards]

@bentael quick update: first draft of the vbexporter has been submitted for open-sourcing approval.

It needs a bit more cleanup for general consumption ( it's my first node_module "release" ) but it's a decent starting point.

Also I'm going to go ahead and fork your importer. Once I've got anything worth contributing back I'll figure out how to issue a pull request ( first time for that, too ).

Thanks for all your work (and the node team too)! I'm excited to be able to contribute something back.

[bentael]

@medwards sweet! once it's opened, I will fork it and give you a hand polishing it up.

[medwards]

@bentael @psychobunny @julian

So I have approval to open source the vb exporter. It's rough at the moment and I won't have time to improve it a for a few weeks, but at least it'll be out there. I'll post a link later today.

Quick question, also. After the most recent run of the importer my "admin" user (uid:1) does not appear to be an admin user. He's in the gid:2:members set which is the administrators group. I found an old post that said to add the id to the "administrators" global key in redis, but that doesn't appear to exist anymore.

Any pointers?

[bentael]

@medwards i may have an idea why, investigating..

[bentael]

/facepalm
https://github.com/akhoury/nodebb-plugin-import/blob/master/lib/import.js#L522 that's a left over from my early nib code.

but that may require a little more work to fix that i thought. If you already migrated, try to make the use an Admin manually.
go to the /admin/users/search route, find that user, click on Make Admin.

[bentael]

oh wait, im dumb, if do not have any admin users, you can't go the to /admin page.

@medwards it's 'group:gid' global object,

redis-cli hgetall group:gid
# you should get an 'adminisitrator' field with value, most likely 2

# then, assuming gid is 2, and the user.id that you want to make admin is also 2 (just a coincidence), so really the command is sadd gid:{gid}:members {uid}
sadd gid:2:members 2

[medwards]

@bentael I think the database is already in the state you are trying to get me to put it in:

redis> hgetall group:gid

1. "registered-users"
2. "1"
3. "administrators"
4. "2"

redis> smembers gid:2:members

1. "1"

redis> hget user:1 username
"admin"

redis>

The user "admin" (uid:1) is already in the gid:2:members set (which is the "administrators" group).

Are we sure the latest version didn't change the way privileges are allocated?
@psychobunny