How to restore topic posters as topic followers?

Juan G.

After importing forums into NodeBB, users are not getting notifications of new replies to topics they have replied to. They are not set as followers/watchers of their topics anymore, excepting the first poster only.

An example of a MongoDB record for a long topic with a large number of posters but only three followers (OP, admin, tester):

{
    "_id" : ObjectId("5ab6da748fda465c4283b0f5"),
    "_key" : "tid:899:followers",
    "members" : [ 
        "4614", 
        "1", 
        "20790"
    ]
}

The follower with uid 4614 is the original poster of the topic. 1 is of course the admin. 20790 is a temporary user to test adding himself manually as watcher of the topic.

Related to this record, there are also a large number of records for the many posters on this long forum topic (e.g., for users 4614, 9623...):

{
    "_id" : ObjectId("5ab6da748fda465c4283b0e5"),
    "_key" : "tid:899:posters",
    "value" : "4614",
    "score" : 8
}

{
    "_id" : ObjectId("5ab6dc328fda465c4292cfff"),
    "_key" : "tid:899:posters",
    "value" : "9623",
    "score" : 1
}

etc...

This happens with all topics and users of the imported forums.

A good way to fix it would be if selecting "Follow topics that you reply to" as default user setting configured existing users as well, since that's usually the expected behavior for forums. However, currently this setting doesn't restore previous topic posters as topic followers, and they don't receive notifications.

Please, is there a way to restore them as followers? Those forums have thousands of users, so we can't manually edit the databases. Maybe some kind of script for MongoDB? Not sure how to fix it.

<baris>

@juan-g Here is a one time script to get the posters of each topic and have them follow that topic. Make sure you take a backup of your database before running this as I haven't tested it on a live database. Hope it helps.

/*globals require, console, process */
'use strict';

var nconf = require('nconf');
var async = require('async');

nconf.file({
	file: 'config.json'
});

nconf.defaults({
	base_dir: __dirname,
});

var db = require('./src/database');

db.init(function(err) {
	if (err) {
		console.log("NodeBB could not connect to your database. Error: " + err.message);
		process.exit();
	}

	followTopics(function (err) {
		if (err) {
			console.error(err);
			process.exit();
		}
		console.log('done');
		process.exit();
	});
});

function followTopics(callback) {
	db.getSortedSetRange('topics:tid', 0, -1, function (err, tids) {
		if (err) {
			return callback(err);
		}
		
		async.eachSeries(tids, followTopicByPosters, callback);
	});
}

function followTopicByPosters(tid, callback) {
	db.getSortedSetRange('tid:' + tid + ':posters', 0, -1, function (err, uids) {
		if (err) {
			return callback(err);
		}

		if (!uids.length) {
			return callback();
		}
                db.setAdd('tid:' + tid + ':followers', uids, callback);
	});
}

You have to place this in your NodeBB root folder and run it with node myScript.js

Juan G.

@baris Wow, thank you, that's amazing. A doubt before running it. As said, the first poster of each topic is already a follower (although not the rest of posters), however I think db.setAdd will prevent duplicate uids when adding them this way, to avoid maybe duplicate notifications for the OPs. But I'm not sure. Is that so? Thanks!

<baris>

@juan-g yes db.setAdd won't add duplicates.

Juan G.

@baris Almost there, this is really wonderful. I did, from the nodebb folder:

./nodebb stop

mongodump -u nodebb -p [password] -d nodebb -o dump/nodebb-2018-05-03

node restoreTopicFollowers.js

Then verified the MongoDB database (I use Robo 3T for that), e.g.:

db.getCollection('objects').find({_key: /:followers/})

and indeed the followers had been correctly added to each topic on the database!

So I did:

./nodebb start

And testing NodeBB there is a final glitch: Those added followers still show "This user hasn't watched any topics yet" in their "Watched" sections. I've tried rebuilding and restarting NodeBB, and deleting browser cache, but no change there.

Is there anything I can try for that?

<baris>

Oh knew I forgot something, run the below to create those sets.

/*globals require, console, process */
'use strict';

var nconf = require('nconf');
var async = require('async');

nconf.file({
	file: 'config.json'
});

nconf.defaults({
	base_dir: __dirname,
});

var db = require('./src/database');

db.init(function(err) {
	if (err) {
		console.log("NodeBB could not connect to your database. Error: " + err.message);
		process.exit();
	}

	followTopics(function (err) {
		if (err) {
			console.error(err);
			process.exit();
		}
		console.log('done');
		process.exit();
	});
});

function followTopics(callback) {
	db.getSortedSetRange('topics:tid', 0, -1, function (err, tids) {
		if (err) {
			return callback(err);
		}

		async.eachSeries(tids, followTopicByPosters, callback);
	});
}

function followTopicByPosters(tid, callback) {
	db.getSortedSetRange('tid:' + tid + ':posters', 0, -1, function (err, uids) {
		if (err) {
			return callback(err);
		}

		if (!uids.length) {
			return callback();
		}

		var keys = uids.map(function (uid) {
			return 'uid:' + uid + ':followed_tids';
		});
		db.sortedSetsAdd(keys, now, tid, callback);
	});
}

Juan G.

@baris It returns an error now is not defined for the line 56:

db.sortedSetsAdd(keys, now, tid, callback);

Juan G.

Maybe I can try using Date.now() instead of now, but I don't know if that's right for this.

Juan G.

This is great. It works!

So, if I'm not wrong, to fix other imported forums we should also run both scripts (restoreTopicFollowers.js and restoreTopicFollowers2.js), the second one with the minor replacement now -> Date.now()

Really, thank you very much!

julian

Yes, typically when you see now used in our code there's probably a var now = Date.now(); nearby

<baris>

@juan-g Yeah you can declare now above the code with var now = Date.now()