Migrating to NodeBB - good stuff

[medwards]

Hi,

I’m an engineer at Maker Studios currently in the process of migrating our old forums from vBulletin over to NodeBB. We’ve got tens of thousands of users, approaching 100k posts. So far things are going well and all the feedback is great, great job! Now that I’ve learned my way around a bit I thought I’d speak up to ask for some input on a few things.

So far I’ve written a node script to dump our vBulletin data in the nodebb-plugin-import format (thanks @bentael ), spun up dev, test and prod environments on Heroku(v0.2.1, upgrade planned), loaded our data using the “plugin” (currently on RedisToGo), and written some code to integrate with our core applications user authentication to auto-create/login users to NodeBB. Another member of my team is working on a custom theme for us, not sure on the timeline on that, though.

Eventually I’m hoping to open source the VB dump script (not that it’s all that complicated) along with contributing back any bug fixes/features that are general enough to be useful to the broader NodeBB community. I’ll need some approvals from higher up for that, but things are looking good.

I’ve been working in a private fork for now, just created a new public fork today to generate pull requests if/when I have changes worth pulling upstream: https://github.com/MakerStudios/NodeBB

I’m about to create a few threads to discuss some issues we’re looking at before launch, any input would be greatly appreciated. I’ll link them from here in a moment.

1. Search - looking into your discussions with @seti123, haven’t dug in quite yet. link
2. Redis vs. Mongo - especially if it impacts search
3. nodebb-plugin-import “issues” (got it working, some notes and a few questions @bentael ) link
4. Converting “vb code” to markdown during data import, anybody have suggestions?
5. Authentication “plugin” - I patched authentication.js to add express middleware(app.use()), but would like to refactor to a plugin if there’s a hook for that? link

Thanks for all your hard work, this stuff is great - keep it coming :).

[julian]

Wow @medwards, awesome stuff!

1. Search is provided by Reds on Redis, and is not exactly what you'd expect. It works on phonetics (so a search for "fonetiks" should bring up this post), and it really confuses some people. @seti123 is working on an elasticsearch plugin, and that might actually help offload some of the storage costs for Redis (since it is essentially stored off-site)
2. Redis (faster, memory based), Mongo (slower, disk based) -- more in-depth available on request
3. Defer to @bentael (akhoury)
4. VB Code is BBcode, no? Perhaps @adarqui's BBCode plugin may be of assistance... this plugin may be quite out of date, needs some love

Keep in mind using BBCode as a parser instead of Markdown would work, although there are parts of the site that assume you are using Markdown.

I'm confused by #5... (think)

I'm sure @bentael would love to have access to your VB dump script.

[bentael]

3. the import plugin was designed for 0.2.0, I was planning to bump it's support to 0.3.1'ish soon, many many things changed and I KNOW it's broken, it's even broken for 0.2.1, im not sure how you did that lol, Now about the rest of bugs, I'm open to more investigation and fixes.
4. converting vb code to markdown, it may be possible during data import, for now I CAN add HTML to Markdown, I know this is not what you wanted to hear, but if you can can find a good NodeJS module to convert Vb Code to Markdown, ill add it's support.

let me ask you this, did you migrate your site yet? and now you're trying to fix after the fact or can you hold that thought, till I bump the import plugin to 0.3.1 ETA this weekend (not 0.3.0 since some signatures changed)

if you can send me a few files of each "model" of your VB dump, I'll test against that.

[medwards]

@bentael We are not live.

I'm debugging the data import process so when it comes time to cut over I can shut down the old forum, do a data dump -> import and update DNS stuff etc. to cut-over.

I'll try to get you some example data before this weekend if that's your timeline to do some patching. I'll need to hand edit to remove content and just leave the skeleton, but happy to do it so you can provide input.

[bentael]

ok, email me at [email protected]
also, if you find a good NodeJS "VB Code to Markdown" I will add a config to choose which format to convert from.

but just to be clear after I patch it, the import plugin will NOT support 0.3.0 and lower, only 0.3.x-edge. Are you willing to wait for 0.3.1 to import again?

[medwards]

@bentael I've heard discussions that we'd like to go live in the next 1-2 weeks and will want some time to confirm I have a stable data migration pipeline.

Not sure what the timeline on 0.3.1 is...?

I guess I should be okay to run edge for now until it get's there, though...

[bentael]

@medwards oh - ok. Looks like 0.3.1 is not due till like a month.

tell you what, I'll patch up to 0.3.0 so you can import to that, then it's just a matter of ./nodebb upgrade from there.

[julian]

@bentael @medwards For the most part, we're limiting refactors to minor version bumps, and as the next version is 0.3.1, you should be fine if @bentael fixes up the importer for edge, even if 0.3.1 isn't released by then.

[bentael]

@julian got you, i actually prefer doing that than falling back to 0.3.0

[bentael]

Sorry a long ass reply with snapshots, it's 5:30am don't judge me.

@medwards

I think nodebb-plugin-import is ready for your set of eyes looking at it, if you can test it out with your real data (didn't publish to npm yet, still running some tests before I do that)

here's what I did:

• Basically made it compatible with 0.3.x (latest commit was 46e29dfb2b841ad4919059886b8c68f1c21da77e), however, and unfortunately, you still need to comment-out/change few lines of code in NodeBB's src, please read this readme section very carefully and click on each image to view the lines numbers to comment. Keep in mind that these line numbers are based on the commit-snapshot i mentioned in the beginning of this long sentence.
• added both html-to-md and bbcode-to-md support to config.convert so you can choose either, and the performance is not that bad 8minutes (mac mini, i5, 16gb) to import that sample dump you sent me, look at the config section in the readme.
• fixed locked issue, now ALL the topics are assumed unlocked with a flag, but then I relock them after all the posts get imported in
• fixed pinned/unpinned issue
• guarded against some the potential Errors you mentioned.

give it a try, let me know, don't forget to revert the NodeBB src changes right after.
Thanks for sending these sample dump files btw, nice work on the automated generation. I couldn't test the bbcode conversion since the dump content were all clear text, but I stole a function from here, then modified a little, see code here, I'll let you test it out.

Also, I am not sure what's the difference between bb Code and vB code, can anyone explain?

#Snapshots:

locking works with a reply posted to it

pinned threads now float to the top

general migrated data looked good

[julian]

@bentael Thanks so much for the work you're putting into this. You are providing an invaluable service to the community

[bentael]

@julian de nada.
hey do you the difference between vB Code and bb code ?

[psychobunny]

wow, awesome @bentael

vB code

vBulletin.org Forum - BB Code List

vs

ubb code
http://www.ubbwiki.com/index.php/UBBCode

imo the one you used for ubb should be more than satisfactory for vB sites (just need to change the : to =)

[bentael]

@psychobunny actually your second link should be http://en.wikipedia.org/wiki/BBCode

UBB used straight HTML, at least that's what mine did.

Anyways, looking at the bbcode vs the vB code, they look the same, to me al least. so the bbcode-to-html conversion should work fine.

[medwards]

@bentael Followed your instructions including code modifications - worked great! Pinned and locked threads are coming through as expected now, and the bbcode conversion sort of worked.

The tags that are included in that regex list were properly replaced - I think I'm going to have to look through that link @psychobunny provided to ubbcode and add a few extra tags: [quote], [center] and maybe a few others, but some of the formatting came across nicely.

Thanks for all the great work! I'll circle back once I get a chance to shore up the code converter to provide my updates. Your work got me to a nice stopping point for now and I'm going to have to prioritize our authentication/user creation stuff so I'll hopefully get back to the bbcode conversion a bit later in the week.

[bentael]

@medwards great to hear ! feel free to edit that RegExp, I accept pull requests - and if it's a new format, we can add that too.

Don't forget to check with your superiors to open-source that "vB exporter" when you have a chance. (name it something like nodebb-plugin-import-vbulletin to keep the convention) - if you don't have the time to write a full read-me and docs, that's fine, I can help out on that.

[medwards]

@bentael quick update: first draft of the vbexporter has been submitted for open-sourcing approval.

It needs a bit more cleanup for general consumption ( it's my first node_module "release" ) but it's a decent starting point.

Also I'm going to go ahead and fork your importer. Once I've got anything worth contributing back I'll figure out how to issue a pull request ( first time for that, too ).

Thanks for all your work (and the node team too)! I'm excited to be able to contribute something back.

[bentael]

@medwards sweet! once it's opened, I will fork it and give you a hand polishing it up.

[medwards]

@bentael @psychobunny @julian

So I have approval to open source the vb exporter. It's rough at the moment and I won't have time to improve it a for a few weeks, but at least it'll be out there. I'll post a link later today.

Quick question, also. After the most recent run of the importer my "admin" user (uid:1) does not appear to be an admin user. He's in the gid:2:members set which is the administrators group. I found an old post that said to add the id to the "administrators" global key in redis, but that doesn't appear to exist anymore.

Any pointers?

[bentael]

@medwards i may have an idea why, investigating..