reset all data via DB modules

bentael

Hey guys, (maybe @barisusakli)

In process of writing v0.2.0 of the nodebb-plugin-import, with UI support in the ACP, and doing major refactoring.

One thing I am a little stuck on, is that I'd like to be able to "restart" the import process if the user wants to do that.
The way I was doing it in 0.1.x is that I that run a "flushDB" command on the DB (only supported redis and mongo), then re-run the node app --setup={...} programatically. This way has been very error prone, only worked on UNIX and is not really possible now that the ./nodebb process needs to be running during the import-process, so I'd like to get away from that.

what I would like to do, something like

async.series([
  db.deleteAllUsers,
  db.deleteAllCategories,
  db.deleteAllTopics,
  db.deleteAllPosts
], db.flushAllIndexes);

I am looking at the db.listRemoveAll(key, value, callback) function, it looks promising but I am not too sure what are the key patterns I should be using without f**king shit up.

if you can point these out for me that would save me a lot of time.

Thanks

bentael

to be clear, I am not asking to add deleteAll* functions to the db module, I can write those in the importer plugin, I just want a list of the keys to delete, along with their corresponding indexes.

<baris>

For categories the sorted set that stores the ids is categories:cid. So to delete all categories

db.getSortedSetRange('categories:cid', 0, -1, function(err, cids){
    deleteAllCategories(cids); //can use categories.purge() internally
});

For users the sorted set is users:joindate, similar to categories.

db.getSortedSetRange('users:joindate', 0, -1, function(err, uids) {
    deleteAllUsers(uids); //can user User.delete internally
});

Now categories.purge also purges topics and posts inside them so this should be enough, but for whatever reason there are left over topics or posts you can do the same for those as well. The sorted set that stores all the topic ids is topics:tid and the sorted set that stores all the post ids is posts:pid.

So to recap the list of sorted sets that store cid, uid, tid and pid respectively are categories:cid, users:joindate, topics:tid and posts:pid.

Let me know if you need anything else.

bentael

awesome! thanks! what about db search indexes? do i have to worry about those?

<baris>

Ahh for those you can use the db.searchRemove functions

db.searchRemove('post', pid);
db.searchRemove('topic', tid);

This only applies to the nodebb-plugin-db-search.

bentael

I see. thanks

<baris>

Side note, if you are resetting all the data you probably need to reset the next ids and counts for users, posts, topics and categories as well. They are in the global hash.

db.setObject('global', {
  nextPid: 0,
  nextTid: 0,
  nextUid: 0,
  nextCid: 0,
  userCount: 0,
  topicCount: 0,
  postCount: 0
});

bentael

@baris thanks!

I don't like to brag but I kinda figured this out 16h ago

I was checking uid==0 and skipping that out from deletion, but kept getting deleted. My admin uid kept incrementing for every node app --setup, so I looked at the schema again.

but thanks your helper methods were extremely helpful.