nodebb-plugin-import - Notes and Questions


  • I was able to get our vBulletin data dump imported using nodebb-plugin-import (thanks @bentael ), but I had to make a few changes to the NodeBB codebase during import to avoid it crashing, as well as a minor fix to the “plugin” itself.

    src/categories.js: added “var meta = require(‘./meta’);” near the top
    src/users.js: commented out line 870 “websockets.in( …etc.”
    https://github.com/akhoury/nodebb-plugin-import/blob/master/lib/import.js
    lines 646-647 need guards like on 537-538

    I also noticed that if I import locked threads as locked, their replies fail to import (makes sense). Also pinned threads are marked as pinned but don’t float to the top of the post listings unless they get unpinned/repinned. If there’s a known fix/work-around for either of these I’d love to hear about it, or I’ll share mine once I find one.

  • Plugin & Theme Dev

    @medwards said:

    src/categories.js: added “var meta = require(‘./meta’);” near the top
    src/users.js: commented out line 870 “websockets.in( …etc.”

    I did comment few more lines actually on the 0.2.0 see this section of the readme

    https://github.com/akhoury/nodebb-plugin-import/blob/master/lib/import.js
    lines 646-647 need guards like on 537-538

    sure

    I also noticed that if I import locked threads as locked, their replies fail to import (makes sense).

    I think you might have to comment a little more to avoid NodeBB restricting replies on locked threads

    Also pinned threads are marked as pinned but don’t float to the top of the post listings unless they get unpinned/repinned. If there’s a known fix/work-around for either of these I’d love to hear about it, or I’ll share mine once I find one.

    if the data is correct in the DB, then it sounds like a NodeBB bug, but I can investigate if you include a variaty of pinned/unpinned/locked/unlocked topics in your sample dump


  • @bentael I think the pinning "issue" occurs because the database is denormalized. When you create a thread the "normal way" through the interface it fires some extra actions that don't occur when you create it directly using the DB wrapper objects. I looked briefly at the code and I think it modifies a sorted set when you create the thread with the interface, but it just creates the hash for the thread when you do it behind the scenes and doesn't touch the Redis key where the ordering is stored.


  • @bentael I saw the section of the readme you refer to. It seemed to primarily be focused on performance. The code modifications I made were actually required to stop the process from crashing, though. Of course, that's not surprising since I was using an unsupported version ;P, I just thought I'd draw attention to it for posterity :).

  • Plugin & Theme Dev

    • pinning issue: i see, but still that doesn't right though, oh well, then we would have to mimic what NodeBB is doing, load the module, then run whatever function. can you pin-point what code path does the pinning? @julian @baris @psychobunny
    • crashing issues: i can't comment without logs, but im glad you got that working.
  • Admin Staff

    https://github.com/designcreateplay/NodeBB/blob/master/src/threadTools.js#L154

    The way it works is, if a thread is pinned it is assigned the largest possible time value so it floats to the top in that sorted set. I guess that function isnt called. Just setting the pinned field on the topic object won't work alone.

  • Plugin & Theme Dev

    ahh, ok.. thanks @baris

    @medwards sounds like an easy fix. I think.. all i need to add is this, not call the whole function, since I already have the cid and I dont care the updating the websockets during the migration

    db.sortedSetAdd('categories:' + cid + ':tid', Math.pow(2, 53), tid);
    
  • Admin Staff

    Yeah I don't like how that function has socket stuff in it that should move to the socket.io folder. That function should be just

    ThreadTools.pin = function(tid, uid, callback) {
    		topics.setTopicField(tid, 'pinned', 1);
    		topics.getTopicField(tid, 'cid', function(err, cid) {
    			db.sortedSetAdd('categories:' + cid + ':tid', Math.pow(2, 53), tid, function(err) {
    				if (callback) {
    					callback(err, {
    						tid: tid
    					});
    				}                        
    			});
    		});
    	}
    

    plus better error checking. same for all the functions in that file.

  • Plugin & Theme Dev

    @baris agree.

    also @medwards about the locked threads, you should comment this line

    Sorry never thought of that, Ill add that to the README and I will include the pinned fix in this weekend's patch


  • @bentael @baris

    Great, that should definitely help lower the workload to clean up this import.

    @bentael I'll get you some of our data tomorrow to test with.

  • Plugin & Theme Dev

    i replied in the wrong thread, well related thread, http://community.nodebb.org/topic/617/#4304

Log in to reply
 

Suggested Topics

| |