Total vote count on topic list

<baris>

@phenomlab I sent a PR to @oplik0 's repro that will change the sorting. https://github.com/oplik0/nodebb-plugin-total-vote-count/pull/19

Just a warning, this will only apply to topics when they receive a new vote on one of their posts. It basically recalculates the new order whenever there is a new vote. Also once this plugin is active uninstalling it will not magically fix the scores to go back to the old sorting. You would need a custom script to fix all the topic votes to set them back to the main post only.

phenomlab

@baris Ok. Many thanks

crazycells

thanks a lot @baris

I Really Appreciate You Curtis Payne GIF

just a quick question, this plugin is not causing any performance issues, right? I am not planning to uninstall it since it is -for us- more correct way to calculate the topic vote counts, but wanted to check since you mentioned some side effects of the plugin after uninstalling it...

<baris>

It might slow down the topic list if there are topics with a huge amount of posts, since it is calculating the total vote count when you load it. You would have to test and see if the performance is acceptable.

crazycells

@baris theoretically, this action will only happen when we try to rank the topics according to votes, right? I mean, if recently replied is picked, should this create a performance issue?

in our forum, most replied topics have anywhere between 20K-30K posts, not more than that...

<baris>

No the code in the plugin runs whenever you load a list of topics to calculate the votes count displayed. Instead of displaying the normal value it is calculating total votes from each post in the topic. So yeah topics with 30k posts might slow things down but I haven't tested it.

phenomlab

@baris my concern at this point is related to performance. Especially if things remain the same even after this specific plugin is disabled or uninstalled.

Will there be an option in the admin console to reset this with the script you previously mentioned?

<baris>

@phenomlab you can run below script in your nodebb folder to reset to the old scores.

/* eslint-disable no-await-in-loop */
/* globals require, console, process */

'use strict';

const nconf = require('nconf');

nconf.file({
	file: 'config.json',
});

nconf.defaults({
	base_dir: __dirname,
	views_dir: './build/public/templates',
	upload_path: 'public/uploads',
});

const db = require('./src/database');

db.init(async (err) => {
	if (err) {
		console.log(`NodeBB could not connect to your database. Error: ${err.message}`);
		process.exit();
	}

	await fixVotes();
	console.log('done');
	process.exit();
});


async function fixVotes() {
        const batch = require('./src/batch');
        await batch.processSortedSet('topics:tid', async (tids) => {
                let topicData = await db.getObjectsFields(
                        tids.map(tid => `topic:${tid}`),
                        ['tid', 'cid', 'timestamp', 'upvotes', 'downvotes']
                );
                topicData = topicData.filter(t => t && t.cid);
                topicData.forEach((t) => {
                        t.votes = parseInt(t.upvotes, 10) - parseInt(t.downvotes, 10);
                });

                await db.sortedSetAddBulk(
                        topicData.map(t => ([`cid:${t.cid}:tids:votes`, t.votes, t.tid]))
                );
                await db.sortedSetAddBulk(
                        topicData.map(t => ([`topic:votes`, t.votes, t.tid]))
                );

                
        }, {
                batch: 500,
        });
}

phenomlab

@baris thanks

crazycells

@baris said in Total vote count on topic list:

No the code in the plugin runs whenever you load a list of topics to calculate the votes count displayed. Instead of displaying the normal value it is calculating total votes from each post in the topic. So yeah topics with 30k posts might slow things down but I haven't tested it.

yes, unfortunately, there is a performance issue... there is several seconds of delay while topics are listed...

phenomlab

@crazycells said in Total vote count on topic list:

there is a performance issue...

Which is why I want to back this out as I'm experiencing a similar issue. I wonder if it's possible to calculate the total votes as a nightly routine which updates a table (for example). That table is referenced on topic list load so topic id = total votes, etc.

A nighttime run that can be specified as off hours would work well in this case. Not realtime but certainly usable. Plus the ability to run the task adhoc in the admin console....

Just a thought.

crazycells

@phenomlab said in Total vote count on topic list:

@crazycells said in Total vote count on topic list:

there is a performance issue...

Which is why I want to back this out as I'm experiencing a similar issue. I wonder if it's possible to calculate the total votes as a nightly routine which updates a table (for example). That table is referenced on topic list load so topic id = total votes, etc.

A nighttime run that can be specified as off hours would work well in this case. Not realtime but certainly usable. Plus the ability to run the task adhoc in the admin console....

Just a thought.

I totally agree with you. I definitely think this is the correct way of calculation topic vote counts...

I believe it is a dynamic function that makes the calculations each time a topic list page is loaded... however it does not have to be that dynamic... It would be great, if what you said can be applied, a list with these variables can be re-calculated every 24 hours... Ultimately, topics are not typically ranked with votes 345, 346 and 347 anyways... instead, they would be something like one with 300s, another one with 400s, one with 600s... So, it does not need to be an exact number at any given moment, daily fluctuations should not significantly affect the rankings...

phenomlab

@baris said in Total vote count on topic list:

Also once this plugin is active uninstalling it will not magically fix the scores to go back to the old sorting

Not the case in my experience. I uninstalled the plugin, and everything returned to normal - I did not need to do anything /top now works as expected, but would prefer this to be the total votes - obviously, unless we can run this as a scheduled task as I previously mentioned, then it's non-starter because of the performance impact.
I did try to run the script, but it didn't work

/home/sudonix.org/nodebb/vote.js:53
                progress: this.progress,
                               ^

TypeError: Cannot read properties of undefined (reading 'progress')
    at fixVotes (/home/sudonix.org/nodebb/vote.js:53:32)
    at /home/sudonix.org/nodebb/vote.js:26:8
    at /home/sudonix.org/nodebb/src/promisify.js:43:65
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

<baris>

@phenomlab I fixed the script in the above post, also made some changes to the plugin so it doesn't run the calculation on every page load. https://github.com/oplik0/nodebb-plugin-total-vote-count/pull/21

It will only calculate the total vote count if it is missing so any loads after that will use the cached value. Hopefully that improves performance.

phenomlab

@baris Great! I'll have a another go with this.

phenomlab

@baris said in Total vote count on topic list:

I fixed the script in the above pos

Still doesn't work for me

throw new Error(`[[error:invalid-score, ${item[1]}]]`);
^

Error: [[error:invalid-score, NaN]]
    at /home/sudonix.org/nodebb/src/database/mongo/sorted/add.js:84:11
    at Array.forEach (<anonymous>)
    at module.sortedSetAddBulk (/home/sudonix.org/nodebb/src/database/mongo/sorted/add.js:82:8)
    at Object.wrapperCallback [as sortedSetAddBulk] (/home/sudonix.org/nodebb/src/promisify.js:46:11)
    at batch.processSortedSet.batch (/home/sudonix.org/nodebb/vote.js:44:26)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async module.processSortedSet (/home/sudonix.org/nodebb/src/database/mongo/sorted.js:602:5)
    at async exports.processSortedSet (/home/sudonix.org/nodebb/src/batch.js:30:10)
    at async fixVotes (/home/sudonix.org/nodebb/vote.js:34:9)
    at async /home/sudonix.org/nodebb/vote.js:26:2

<baris>

@phenomlab maybe try with below

/* eslint-disable no-await-in-loop */
/* globals require, console, process */

'use strict';
 
const nconf = require('nconf');
 
nconf.file({
	file: 'config.json',
});
 
nconf.defaults({
	base_dir: __dirname,
	views_dir: './build/public/templates',
	upload_path: 'public/uploads',
});
 
const db = require('./src/database');
 
db.init(async (err) => {
	if (err) {
		console.log(`NodeBB could not connect to your database. Error: ${err.message}`);
		process.exit();
	}
 
	await fixVotes();
	console.log('done');
	process.exit();
});
 
 
async function fixVotes() {
	const batch = require('./src/batch');
	await batch.processSortedSet('topics:tid', async (tids) => {
		let topicData = await db.getObjectsFields(
			tids.map(tid => `topic:${tid}`),
			['tid', 'cid', 'timestamp', 'upvotes', 'downvotes']
		);
		topicData = topicData.filter(t => t && t.cid);
		topicData.forEach((t) => {
			t.votes = parseInt(t.upvotes || 0, 10) - parseInt(t.downvotes || 0, 10);
		});
 
		await db.sortedSetAddBulk(
			topicData.map(t => ([`cid:${t.cid}:tids:votes`, t.votes, t.tid]))
		);
		await db.sortedSetAddBulk(
			topicData.map(t => ([`topic:votes`, t.votes, t.tid]))
		);
 
		
	}, {
		batch: 500,
	});
}

phenomlab

@baris that seems to work.

phenomlab

@baris I've installed the latest version of this plugin, and have new settings as below

These seem to work fine, but when using /top the votes are not in any specific order?

<baris>

@phenomlab what is the url on the page it doesn't work? Is it just /top or does it have any query params?