nodebb-plugin-import - Notes and Questions

medwards

I was able to get our vBulletin data dump imported using nodebb-plugin-import (thanks @bentael ), but I had to make a few changes to the NodeBB codebase during import to avoid it crashing, as well as a minor fix to the “plugin” itself.

src/categories.js: added “var meta = require(‘./meta’);” near the top
src/users.js: commented out line 870 “websockets.in( …etc.”
https://github.com/akhoury/nodebb-plugin-import/blob/master/lib/import.js
lines 646-647 need guards like on 537-538

I also noticed that if I import locked threads as locked, their replies fail to import (makes sense). Also pinned threads are marked as pinned but don’t float to the top of the post listings unless they get unpinned/repinned. If there’s a known fix/work-around for either of these I’d love to hear about it, or I’ll share mine once I find one.

bentael

@medwards said:

src/categories.js: added “var meta = require(‘./meta’);” near the top
src/users.js: commented out line 870 “websockets.in( …etc.”

I did comment few more lines actually on the 0.2.0 see this section of the readme

https://github.com/akhoury/nodebb-plugin-import/blob/master/lib/import.js
lines 646-647 need guards like on 537-538

sure

I also noticed that if I import locked threads as locked, their replies fail to import (makes sense).

I think you might have to comment a little more to avoid NodeBB restricting replies on locked threads

Also pinned threads are marked as pinned but don’t float to the top of the post listings unless they get unpinned/repinned. If there’s a known fix/work-around for either of these I’d love to hear about it, or I’ll share mine once I find one.

if the data is correct in the DB, then it sounds like a NodeBB bug, but I can investigate if you include a variaty of pinned/unpinned/locked/unlocked topics in your sample dump

medwards

@bentael I think the pinning "issue" occurs because the database is denormalized. When you create a thread the "normal way" through the interface it fires some extra actions that don't occur when you create it directly using the DB wrapper objects. I looked briefly at the code and I think it modifies a sorted set when you create the thread with the interface, but it just creates the hash for the thread when you do it behind the scenes and doesn't touch the Redis key where the ordering is stored.

medwards

@bentael I saw the section of the readme you refer to. It seemed to primarily be focused on performance. The code modifications I made were actually required to stop the process from crashing, though. Of course, that's not surprising since I was using an unsupported version ;P, I just thought I'd draw attention to it for posterity :).

bentael

pinning issue: i see, but still that doesn't right though, oh well, then we would have to mimic what NodeBB is doing, load the module, then run whatever function. can you pin-point what code path does the pinning? @julian @baris @psychobunny
crashing issues: i can't comment without logs, but im glad you got that working.

<baris>

File not found · NodeBB/NodeBB

Node.js based forum software built for the modern web - File not found · NodeBB/NodeBB

GitHub (github.com)

The way it works is, if a thread is pinned it is assigned the largest possible time value so it floats to the top in that sorted set. I guess that function isnt called. Just setting the pinned field on the topic object won't work alone.

bentael

ahh, ok.. thanks @baris

@medwards sounds like an easy fix. I think.. all i need to add is this, not call the whole function, since I already have the cid and I dont care the updating the websockets during the migration

db.sortedSetAdd('categories:' + cid + ':tid', Math.pow(2, 53), tid);

<baris>

Yeah I don't like how that function has socket stuff in it that should move to the socket.io folder. That function should be just

ThreadTools.pin = function(tid, uid, callback) {
		topics.setTopicField(tid, 'pinned', 1);
		topics.getTopicField(tid, 'cid', function(err, cid) {
			db.sortedSetAdd('categories:' + cid + ':tid', Math.pow(2, 53), tid, function(err) {
				if (callback) {
					callback(err, {
						tid: tid
					});
				}                        
			});
		});
	}

plus better error checking. same for all the functions in that file.

bentael

@baris agree.

also @medwards about the locked threads, you should comment this line

Sorry never thought of that, Ill add that to the README and I will include the pinned fix in this weekend's patch

medwards

@bentael @baris

Great, that should definitely help lower the workload to clean up this import.

@bentael I'll get you some of our data tomorrow to test with.

bentael

i replied in the wrong thread, well related thread, http://community.nodebb.org/topic/617/#4304