Hi,

I’m an engineer at Maker Studios currently in the process of migrating our old forums from vBulletin over to NodeBB. We’ve got tens of thousands of users, approaching 100k posts. So far things are going well and all the feedback is great, great job! Now that I’ve learned my way around a bit I thought I’d speak up to ask for some input on a few things.

So far I’ve written a node script to dump our vBulletin data in the nodebb-plugin-import format (thanks @bentael ), spun up dev, test and prod environments on Heroku(v0.2.1, upgrade planned), loaded our data using the “plugin” (currently on RedisToGo), and written some code to integrate with our core applications user authentication to auto-create/login users to NodeBB. Another member of my team is working on a custom theme for us, not sure on the timeline on that, though.

Eventually I’m hoping to open source the VB dump script (not that it’s all that complicated) along with contributing back any bug fixes/features that are general enough to be useful to the broader NodeBB community. I’ll need some approvals from higher up for that, but things are looking good.

I’ve been working in a private fork for now, just created a new public fork today to generate pull requests if/when I have changes worth pulling upstream: https://github.com/MakerStudios/NodeBB

I’m about to create a few threads to discuss some issues we’re looking at before launch, any input would be greatly appreciated. I’ll link them from here in a moment.

1. Search - looking into your discussions with @seti123, haven’t dug in quite yet. link
2. Redis vs. Mongo - especially if it impacts search
3. nodebb-plugin-import “issues” (got it working, some notes and a few questions @bentael ) link
4. Converting “vb code” to markdown during data import, anybody have suggestions?
5. Authentication “plugin” - I patched authentication.js to add express middleware(app.use()), but would like to refactor to a plugin if there’s a hook for that? link

Thanks for all your hard work, this stuff is great - keep it coming :).

@bentael I think the database is already in the state you are trying to get me to put it in:

redis> hgetall group:gid

1. "registered-users"
2. "1"
3. "administrators"
4. "2"

redis> smembers gid:2:members

1. "1"

redis> hget user:1 username
"admin"

redis>

The user "admin" (uid:1) is already in the gid:2:members set (which is the "administrators" group).

Are we sure the latest version didn't change the way privileges are allocated?
@psychobunny

Is there any straightforward way to determine what versions of NodeBB a particular version of a plugin is compatible with? Or vice-versa?

I'm still running a roughly v0.3.2 version of NodeBB and when I do a clean "npm install" I get a version of the nodebb-widget-essentials plugin that crashes the process.

@baris

That does appear to resolve the issue. One issue it brings up, though, is that I have customizations to the lavender theme and the version I've been using appears incompatible with Edge.

Is there any way to identify which versions of the lavender theme are compatible with which versions of NodeBB?

@baris

I'm currently running a patched version of 0.3.2 I think.

I'll try to upgrade and see if it fixes it.

Looking for some input/feedback.

I'm running NodeBB on Heroku and RedisToGo. After a large data import (tens of thousands of posts) when I log in as any user and click into "Unread" and "Mark all as read" the memory usage on both the server and Redis instance spikes. In my case I'm still running in test mode with just 512MB in redis (it usually sits at 309 used or so) and 512MB of memory on a basic Heroku dyno.

Basically the dyno grinds to a halt from swapping and the redis sends an alert about being over quota. Once I kill the dyno memory in Redis drops back down, and of course the restarted dyno doesn't resume the operation.

Is this behavior expected? Do I just need to allocate more memory to allow the operation to complete without blowing out my resource quotas? Is there any way to just flag all posts as read by all users as a starting point since I'm guessing it's trying to mark tens of thousands of posts as read all in one operation?

Thanks!

Sorry for the late reply.

So it sounds like it's not really intended for the HTML Entities to ever make it into the database. I was able to pull a node module "html-entities" and feed the titles through that on the way in through the importer. The post contents were rendering okay so I just left those as is, didn't want to mess around with what order to apply conversions and whether they'd screw each other up.

One of these days I'll get around to contributing back some changes to nodebb-plugin-import. Things have just been crazy around here lately.

Thanks for all your help.

I'm running a data import using @bentael 's importer.

It looks like posts that have escape sequences render correctly as part of a posts content, but display the literal escape sequence in the post title. Screenshot:

And looking at the data via redis on the backend:

127.0.0.1:6379> hget topic:3369 title
"Исчезли деньги ?!"
127.0.0.1:6379> hget post:3369 content
"Всем привет, буквально 5.06 поменял реквизиты с помощью своего мененжера, сегодня захожу в dashboard, а на баллансе 2 доллара, было до этого почти 50 долларов, в разделе Royalties нету выплат, где тогда деньги?"
127.0.0.1:6379> 

It looks like the content and title are being encoded the same way, but for some reason it renders properly in the post content but not the title.

I also tried copying and pasting some of the properly rendered characters as the title of a new post. This appears to work but produces a different encoding within the database:

127.0.0.1:6379> hget topic:3472 title
"\xd0\x92\xd1\x81\xd0\xb5\xd0\xbc \xd0\xbf\xd1\x80\xd0\xb8\xd0\xb2\xd0\xb5\xd1\x82"
127.0.0.1:6379>

Any suggestions? I suppose I could try to perform the conversion between encodings during the import... any idea the specific names of these two encoding types?

@julian So I forked and installed @seti123 's plugin. I had to make a few changes to the way it calls topics.getTopicsByTids() - it looks like the function signature may have changed.

At this point I appear to be getting legitimate search results from the long-form search url that's being installed:

http://localhost:8252/nodebb-plugins-elasticsearch/search/:term

I believe the standing question is:

Is there some way to override the built-in /api/search/:term route so the UI will process the elasticsearch results instead of the built-in REDS results?

I'll dive in and see if I can figure it out... let me know if you have any input.

Thanks!

@julian That was it.

I had an older version of the Lavender theme with the old style admin template.

Thanks!

Are templates cached in some special way that doesn't get cleared on a standard browser refresh? Local storage or something?

I've tried refreshing and restarting the server and the admin cog is still showing up for every user. I can see that I'm loading a stale version of header.tpl because the cog <li> looks like:

<li class="{adminDisplay}">

And I can see the commit that removed that.

Hmm. I guess I can't delete the thread and now having deleted the anchor post leaves this thread pretty garbled.

For anyone else reading this was a silly browser caching thing, not relevant.