Total vote count on topic list

phenomlab

crazycells

@baris said in Total vote count on topic list:

No the code in the plugin runs whenever you load a list of topics to calculate the votes count displayed. Instead of displaying the normal value it is calculating total votes from each post in the topic. So yeah topics with 30k posts might slow things down but I haven't tested it.

yes, unfortunately, there is a performance issue... there is several seconds of delay while topics are listed...

phenomlab

@crazycells said in Total vote count on topic list:

there is a performance issue...

Which is why I want to back this out as I'm experiencing a similar issue. I wonder if it's possible to calculate the total votes as a nightly routine which updates a table (for example). That table is referenced on topic list load so topic id = total votes, etc.

A nighttime run that can be specified as off hours would work well in this case. Not realtime but certainly usable. Plus the ability to run the task adhoc in the admin console....

Just a thought.

crazycells

@phenomlab said in Total vote count on topic list:

@crazycells said in Total vote count on topic list:

there is a performance issue...

Which is why I want to back this out as I'm experiencing a similar issue. I wonder if it's possible to calculate the total votes as a nightly routine which updates a table (for example). That table is referenced on topic list load so topic id = total votes, etc.

A nighttime run that can be specified as off hours would work well in this case. Not realtime but certainly usable. Plus the ability to run the task adhoc in the admin console....

Just a thought.

I totally agree with you. I definitely think this is the correct way of calculation topic vote counts...

I believe it is a dynamic function that makes the calculations each time a topic list page is loaded... however it does not have to be that dynamic... It would be great, if what you said can be applied, a list with these variables can be re-calculated every 24 hours... Ultimately, topics are not typically ranked with votes 345, 346 and 347 anyways... instead, they would be something like one with 300s, another one with 400s, one with 600s... So, it does not need to be an exact number at any given moment, daily fluctuations should not significantly affect the rankings...

phenomlab

@baris said in Total vote count on topic list:

Also once this plugin is active uninstalling it will not magically fix the scores to go back to the old sorting

Not the case in my experience. I uninstalled the plugin, and everything returned to normal - I did not need to do anything /top now works as expected, but would prefer this to be the total votes - obviously, unless we can run this as a scheduled task as I previously mentioned, then it's non-starter because of the performance impact.
I did try to run the script, but it didn't work

/home/sudonix.org/nodebb/vote.js:53
                progress: this.progress,
                               ^

TypeError: Cannot read properties of undefined (reading 'progress')
    at fixVotes (/home/sudonix.org/nodebb/vote.js:53:32)
    at /home/sudonix.org/nodebb/vote.js:26:8
    at /home/sudonix.org/nodebb/src/promisify.js:43:65
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

<baris>

@phenomlab I fixed the script in the above post, also made some changes to the plugin so it doesn't run the calculation on every page load. https://github.com/oplik0/nodebb-plugin-total-vote-count/pull/21

It will only calculate the total vote count if it is missing so any loads after that will use the cached value. Hopefully that improves performance.

phenomlab

@baris Great! I'll have a another go with this.

phenomlab

@baris said in Total vote count on topic list:

I fixed the script in the above pos

Still doesn't work for me

throw new Error(`[[error:invalid-score, ${item[1]}]]`);
^

Error: [[error:invalid-score, NaN]]
    at /home/sudonix.org/nodebb/src/database/mongo/sorted/add.js:84:11
    at Array.forEach (<anonymous>)
    at module.sortedSetAddBulk (/home/sudonix.org/nodebb/src/database/mongo/sorted/add.js:82:8)
    at Object.wrapperCallback [as sortedSetAddBulk] (/home/sudonix.org/nodebb/src/promisify.js:46:11)
    at batch.processSortedSet.batch (/home/sudonix.org/nodebb/vote.js:44:26)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async module.processSortedSet (/home/sudonix.org/nodebb/src/database/mongo/sorted.js:602:5)
    at async exports.processSortedSet (/home/sudonix.org/nodebb/src/batch.js:30:10)
    at async fixVotes (/home/sudonix.org/nodebb/vote.js:34:9)
    at async /home/sudonix.org/nodebb/vote.js:26:2

<baris>

@phenomlab maybe try with below

/* eslint-disable no-await-in-loop */
/* globals require, console, process */

'use strict';
 
const nconf = require('nconf');
 
nconf.file({
	file: 'config.json',
});
 
nconf.defaults({
	base_dir: __dirname,
	views_dir: './build/public/templates',
	upload_path: 'public/uploads',
});
 
const db = require('./src/database');
 
db.init(async (err) => {
	if (err) {
		console.log(`NodeBB could not connect to your database. Error: ${err.message}`);
		process.exit();
	}
 
	await fixVotes();
	console.log('done');
	process.exit();
});
 
 
async function fixVotes() {
	const batch = require('./src/batch');
	await batch.processSortedSet('topics:tid', async (tids) => {
		let topicData = await db.getObjectsFields(
			tids.map(tid => `topic:${tid}`),
			['tid', 'cid', 'timestamp', 'upvotes', 'downvotes']
		);
		topicData = topicData.filter(t => t && t.cid);
		topicData.forEach((t) => {
			t.votes = parseInt(t.upvotes || 0, 10) - parseInt(t.downvotes || 0, 10);
		});
 
		await db.sortedSetAddBulk(
			topicData.map(t => ([`cid:${t.cid}:tids:votes`, t.votes, t.tid]))
		);
		await db.sortedSetAddBulk(
			topicData.map(t => ([`topic:votes`, t.votes, t.tid]))
		);
 
		
	}, {
		batch: 500,
	});
}

phenomlab

@baris that seems to work.

phenomlab

@baris I've installed the latest version of this plugin, and have new settings as below

These seem to work fine, but when using /top the votes are not in any specific order?

<baris>

@phenomlab what is the url on the page it doesn't work? Is it just /top or does it have any query params?

phenomlab

@baris Just /top. For example, https://sudonix.org/top

EDIT - having thought about it, /top doesn't display in vote order by default, but it doesn't display in view order either?

crazycells

@phenomlab I thought /top puts the topics in order according their vote number (number of votes of the first post)...

@baris is this plugin affecting how /top behaves? or just how /popular behaves?

<baris>

Yeah it should display in vote order on /top, it doesn't affect /popular, I have a fix for it in https://github.com/oplik0/nodebb-plugin-total-vote-count/pull/24 but it will require 3.7.2 which will come tomorrow.

phenomlab

@baris Many thanks

phenomlab

@baris updated to 3.7.2 and now just waiting for the plugin to be updated.