@julian I will admit that I changed the thresholds for the 503 errors a while ago because they were happening a ton...
0_1504066715468_4d769fb7-fd00-444b-91f6-3144667a99c7-image.png
Hi,
I’m an engineer at Maker Studios currently in the process of migrating our old forums from vBulletin over to NodeBB. We’ve got tens of thousands of users, approaching 100k posts. So far things are going well and all the feedback is great, great job! Now that I’ve learned my way around a bit I thought I’d speak up to ask for some input on a few things.
So far I’ve written a node script to dump our vBulletin data in the nodebb-plugin-import format (thanks @bentael ), spun up dev, test and prod environments on Heroku(v0.2.1, upgrade planned), loaded our data using the “plugin” (currently on RedisToGo), and written some code to integrate with our core applications user authentication to auto-create/login users to NodeBB. Another member of my team is working on a custom theme for us, not sure on the timeline on that, though.
Eventually I’m hoping to open source the VB dump script (not that it’s all that complicated) along with contributing back any bug fixes/features that are general enough to be useful to the broader NodeBB community. I’ll need some approvals from higher up for that, but things are looking good.
I’ve been working in a private fork for now, just created a new public fork today to generate pull requests if/when I have changes worth pulling upstream: https://github.com/MakerStudios/NodeBB
I’m about to create a few threads to discuss some issues we’re looking at before launch, any input would be greatly appreciated. I’ll link them from here in a moment.
Thanks for all your hard work, this stuff is great - keep it coming :).
Wow @medwards, awesome stuff!
Keep in mind using BBCode as a parser instead of Markdown would work, although there are parts of the site that assume you are using Markdown.
I'm confused by #5... (think)
I'm sure @bentael would love to have access to your VB dump script.
let me ask you this, did you migrate your site yet? and now you're trying to fix after the fact or can you hold that thought, till I bump the import plugin to 0.3.1 ETA this weekend (not 0.3.0 since some signatures changed)
if you can send me a few files of each "model" of your VB dump, I'll test against that.
@bentael We are not live.
I'm debugging the data import process so when it comes time to cut over I can shut down the old forum, do a data dump -> import and update DNS stuff etc. to cut-over.
I'll try to get you some example data before this weekend if that's your timeline to do some patching. I'll need to hand edit to remove content and just leave the skeleton, but happy to do it so you can provide input.
ok, email me at [email protected]
also, if you find a good NodeJS "VB Code to Markdown" I will add a config to choose which format to convert from.
but just to be clear after I patch it, the import plugin will NOT support 0.3.0 and lower, only 0.3.x-edge. Are you willing to wait for 0.3.1 to import again?
@bentael I've heard discussions that we'd like to go live in the next 1-2 weeks and will want some time to confirm I have a stable data migration pipeline.
Not sure what the timeline on 0.3.1 is...?
I guess I should be okay to run edge for now until it get's there, though...
@julian got you, i actually prefer doing that than falling back to 0.3.0
Sorry a long ass reply with snapshots, it's 5:30am don't judge me.
I think nodebb-plugin-import is ready for your set of eyes looking at it, if you can test it out with your real data (didn't publish to npm
yet, still running some tests before I do that)
here's what I did:
html-to-md
and bbcode-to-md
support to config.convert
so you can choose either, and the performance is not that bad 8minutes (mac mini, i5, 16gb) to import that sample dump you sent me, look at the config section in the readme.locked
issue, now ALL the topics are assumed unlocked with a flag, but then I relock them after all the posts get imported inpinned/unpinned
issuegive it a try, let me know, don't forget to revert the NodeBB src changes right after.
Thanks for sending these sample dump files btw, nice work on the automated generation. I couldn't test the bbcode conversion since the dump content were all clear text, but I stole a function from here, then modified a little, see code here, I'll let you test it out.
Also, I am not sure what's the difference between bb Code and vB code, can anyone explain?
#Snapshots:
@bentael Thanks so much for the work you're putting into this. You are providing an invaluable service to the community
@julian de nada.
hey do you the difference between vB Code and bb code ?
wow, awesome @bentael
vB code
vs
ubb code
http://www.ubbwiki.com/index.php/UBBCode
imo the one you used for ubb should be more than satisfactory for vB sites (just need to change the :
to =
)
@psychobunny actually your second link should be http://en.wikipedia.org/wiki/BBCode
UBB used straight HTML, at least that's what mine did.
Anyways, looking at the bbcode vs the vB code, they look the same, to me al least. so the bbcode-to-html
conversion should work fine.
@bentael Followed your instructions including code modifications - worked great! Pinned and locked threads are coming through as expected now, and the bbcode conversion sort of worked.
The tags that are included in that regex list were properly replaced - I think I'm going to have to look through that link @psychobunny provided to ubbcode and add a few extra tags: [quote], [center] and maybe a few others, but some of the formatting came across nicely.
Thanks for all the great work! I'll circle back once I get a chance to shore up the code converter to provide my updates. Your work got me to a nice stopping point for now and I'm going to have to prioritize our authentication/user creation stuff so I'll hopefully get back to the bbcode conversion a bit later in the week.
@medwards great to hear ! feel free to edit that RegExp, I accept pull requests - and if it's a new format, we can add that too.
Don't forget to check with your superiors to open-source that "vB exporter" when you have a chance. (name it something like nodebb-plugin-import-vbulletin
to keep the convention) - if you don't have the time to write a full read-me and docs, that's fine, I can help out on that.
@bentael quick update: first draft of the vbexporter has been submitted for open-sourcing approval.
It needs a bit more cleanup for general consumption ( it's my first node_module "release" ) but it's a decent starting point.
Also I'm going to go ahead and fork your importer. Once I've got anything worth contributing back I'll figure out how to issue a pull request ( first time for that, too ).
Thanks for all your work (and the node team too)! I'm excited to be able to contribute something back.
@medwards sweet! once it's opened, I will fork it and give you a hand polishing it up.
So I have approval to open source the vb exporter. It's rough at the moment and I won't have time to improve it a for a few weeks, but at least it'll be out there. I'll post a link later today.
Quick question, also. After the most recent run of the importer my "admin" user (uid:1) does not appear to be an admin user. He's in the gid:2:members set which is the administrators group. I found an old post that said to add the id to the "administrators" global key in redis, but that doesn't appear to exist anymore.
Any pointers?
@medwards i may have an idea why, investigating..