reset all data via DB modules

NodeBB Development
  • Hey guys, (maybe @barisusakli)

    In process of writing v0.2.0 of the nodebb-plugin-import, with UI support in the ACP, and doing major refactoring.

    One thing I am a little stuck on, is that I'd like to be able to "restart" the import process if the user wants to do that.
    The way I was doing it in 0.1.x is that I that run a "flushDB" command on the DB (only supported redis and mongo), then re-run the node app --setup={...} programatically. This way has been very error prone, only worked on UNIX and is not really possible now that the ./nodebb process needs to be running during the import-process, so I'd like to get away from that.

    what I would like to do, something like

    async.series([
      db.deleteAllUsers,
      db.deleteAllCategories,
      db.deleteAllTopics,
      db.deleteAllPosts
    ], db.flushAllIndexes);
    

    I am looking at the db.listRemoveAll(key, value, callback) function, it looks promising but I am not too sure what are the key patterns I should be using without f**king shit up.

    if you can point these out for me that would save me a lot of time.

    Thanks

  • to be clear, I am not asking to add deleteAll* functions to the db module, I can write those in the importer plugin, I just want a list of the keys to delete, along with their corresponding indexes.

  • For categories the sorted set that stores the ids is categories:cid. So to delete all categories

    db.getSortedSetRange('categories:cid', 0, -1, function(err, cids){
        deleteAllCategories(cids); //can use categories.purge() internally
    });
    

    For users the sorted set is users:joindate, similar to categories.

    db.getSortedSetRange('users:joindate', 0, -1, function(err, uids) {
        deleteAllUsers(uids); //can user User.delete internally
    });
    

    Now categories.purge also purges topics and posts inside them so this should be enough, but for whatever reason there are left over topics or posts you can do the same for those as well. The sorted set that stores all the topic ids is topics:tid and the sorted set that stores all the post ids is posts:pid.

    So to recap the list of sorted sets that store cid, uid, tid and pid respectively are categories:cid, users:joindate, topics:tid and posts:pid.

    Let me know if you need anything else.

  • awesome! thanks! what about db search indexes? do i have to worry about those?

  • Ahh for those you can use the db.searchRemove functions

    db.searchRemove('post', pid);
    db.searchRemove('topic', tid);
    

    This only applies to the nodebb-plugin-db-search.

  • I see. thanks

  • Side note, if you are resetting all the data you probably need to reset the next ids and counts for users, posts, topics and categories as well. They are in the global hash.

    db.setObject('global', {
      nextPid: 0,
      nextTid: 0,
      nextUid: 0,
      nextCid: 0,
      userCount: 0,
      topicCount: 0,
      postCount: 0
    });
    
  • @baris thanks!

    I don't like to brag 😛 but I kinda figured this out 16h ago

    I was checking uid==0 and skipping that out from deletion, but kept getting deleted. My admin uid kept incrementing for every node app --setup, so I looked at the schema again.

    but thanks your helper methods were extremely helpful.


Suggested Topics


  • Upload files via API

    NodeBB Development
    3
    0 Votes
    3 Posts
    344 Views

    Checking logs after an image upload: /api/post/upload might be the droid yer' looking for?

  • 1 Votes
    3 Posts
    417 Views

    @baris yeah it's true, but you can use transaction on a multicluster with only one node, also if with a lot of limitations.
    About your command example is a good idea also if it would work only with redis and postgre, with mongodb can be used like a normal batch.

    I took a look to postgreSQL module, seems it support transactions, mongodb emulate the same thing with non ACID batch. I prefer to avoid SQL databases but the way data is organized seem more suitable to a relational DB

  • Reset all user avatars

    NodeBB Development
    4
    0 Votes
    4 Posts
    877 Views

    What on earth. I bookmarked it now. No clue how or why this got made lol.
    You can purge it.

  • 1 Votes
    4 Posts
    5k Views

    @bentael After some pause, I think I'm ready to work on custom importer. After some investigation I have found, that Importer works with classic/basic entities. Even If I will create custom importer, for example nodebb-plugin-import-ipboard-very-custom, import plugin will use only designed methods (I will be able to import additional fields) like: setup, getPaginatedUsers, getPaginatedCategories, getPaginatedTopics, getPaginatedPosts, and several utility methods like logging, testRun, etc.

    My question: as author of importer, what do you recommend to do if I want to import additional entities? (Example: awards)

    What I want to accomplish after import: Import of standard data: topics, categories, posts, users - It's simple and clear Import custom user fields without altering User document for future plugin use Import additional entity - Awards for future plugin use Import additional entity - Points(like reputation, but isn't part of user table) for future plugin use
  • 0 Votes
    4 Posts
    2k Views

    I don't think we have that documentation right now, couple months ago I started a doc in the github wiki about the db structure but I don't think it got moved over to the docs website.

    Either way the best way to store your plugin data is to prefix the name of your keys with the plugin id so it doesn't conflict with other plugins.

    Also you need to be careful to not break any of the core objects, ie if you delete the key users:joindate you will have problems.