How to improve popular posts response time?

<baris>

Indices look OK as well. Are you running any custom plugins? Maybe one of them is adding the latency. Your cpu usage and database stats look fine. The object cache hit rate looks low. I remember you mentioned a really low group cache hit rate at https://community.nodebb.org/topic/15398/help-fine-tuning-advanced-cache-settings so wondering if a plugin is causing that as well.

What is the output of ./nodebb plugins and is your forum public so we can check?

patsanch

Here's the output

ubuntu@ip-10-111-250-13:~/nodebb$ ./nodebb plugins
2023-02-23T18:44:33.214Z [4567,4568,4569,4570,4571,4572,4573,4574/11855] - verbose: [minifier] utilizing a maximum of 7 additional threads
Active plugins:
	* [email protected] (installed, enabled)
	* [email protected] (installed, enabled)
	* [email protected] (installed, enabled)
	* [email protected] (installed, enabled)
	* [email protected] (installed, enabled)
	* [email protected] (installed, enabled)
	* [email protected] (installed, enabled)
	* [email protected] (installed, enabled)
	* [email protected] (installed, disabled)
	* [email protected] (installed, enabled)
	* [email protected] (installed, disabled)
	* [email protected] (installed, disabled)
	* [email protected] (installed, disabled)
	* [email protected] (installed, enabled)
	* [email protected] (installed, disabled)
	* [email protected] (installed, enabled)

Yup, it is public. You can access here: https://forums.theshow.com

Thanks!

patsanch

@baris said in How to improve popular posts response time?:

Help Fine Tuning Advanced Cache Settings

Hi, I'm on the advanced cache page and I'm seeing Post Cache, Object Cache, Group Cache and Local Cache are all maxed out. I'm guessing this affects performa...

NodeBB Community (community.nodebb.org)

Yep, I think we solved that by deleting unused groups that we set up. We used to have group specific categories. We did that a couple of years and there's a bunch that we don't need anymore. The hit ratio is pretty decent now.

<baris>

[email protected] does this have any custom code? Or is it just css/html changes to slick theme?

https://forums.theshow.com/popular?cid=15 is super slow, but if I use a cid that doesn't exist it's super fast.

I have a feeling there is some custom code that runs for each topic item to make some external requests or such that slows it down when there are topics to display. Someone would have to debug it by placing some logs into the popular route to figure out where exactly it is spending 5secs.

<baris>

One custom data that I see in topics data is postedInByDeveloper: false which plugin is adding this? It is not something added by core.

patsanch

Yes, that is a custom theme based on slick theme.

postedInByDeveloper is something we added to check if a topic has any developer posts.

@Teemberland added that a while ago: https://community.nodebb.org/topic/14867/display-group-badge-on-topic-list/11

patsanch

So it sounds like to fix the performance issue, we will have to remove that hook. Is there no other way to do that in a performant manner?

<baris>

Yeah that custom code could explain the slowdown of popular. Best way to check is to disable it temporarily and see if the pages load fast.
Can simply comment out the code and return immediately and restart nodebb to test.

myPlugin.filterTopicsGet = async function (hookData) {
	// const groupData = await Promise.all(hookData.topics.map(async (t) => {
	// 	 const uids = await db.getSortedSetMembers('tid:' + t.tid + ':posters');
	// 	 const userGroups = await Groups.getUserGroupMembership('groups:createtime', uids);       
	// 	 return userGroups.flat().includes('My developer group');
	// }));
  
	// hookData.topics.forEach((t, i) => {
	// 	t.postedInByDeveloper = groupData[i];
	// });
	return hookData;
  };

If pages are fast then gotto find a more efficient way to show the group badge. That code also explains why the hit rate was low for groups since its calling getUserGroupMembership alot.

<baris>

Yes another way to set postedInByDeveloper would be to do it when someone posts in a topic. In the hook, action:topic.post You can execute that function you have in the hook and set postedByInDeveloper that way it is not called 20 times for each topic when loading topic lists.

patsanch

It's lightning fast without it

<baris>

It would look something like this

myPlugin.actionTopicReply = async function (hookData) {
	const userGroups = await Groups.getUserGroupMembership('groups:visible:createtime', [hookData.data.uid]);       	
	const isInDevGroup = userGroups.flat().includes('My developer group');
	if (isInDevGroup) {
		await topics.setTopicField(hookData.topic.tid, 'postedInByDeveloper', 1);
	}
};

So essentially you save a field into the topic object if a developer posts in it. This way you don't need to calculate it each time the topic list is loaded.

patsanch

Awesome! Thank you so much! We'll try that.

patsanch

This won't retro, but you're right, calculating on post is better than calculating on fetch for each.

<baris>

You might want to use both action:topic.reply and action:topic.post one is triggered on reply and one is triggered when a new topic is posted. Also if the developer group is visible then you can use const userGroups = await Groups.getUserGroupMembership('groups:visible:createtime', [hookData.data.uid]); and it will be even faster.

patsanch

Yes, the group is visible. I'll try the updated snippet.

I'll also add it to both hooks.

Thank you so much!

patsanch

Woot! It's working on our development environment. Only change I made from the snippet was

hookData.topic.tid to hookData.post.topic.tid

Thanks for your help!

<baris>

No problem

patsanch

Just wanted to add what a huge difference our current JMeter tests is telling us. Now we can bump the number of threads to see how much our current set up can handle.

Thank you!

<baris>

I just realized there is a much simpler way to do this, since you only seem to care about if the user is in a special dev group you don't need to load the user's groups, you can just check if they are in that special group.

myPlugin.actionTopicReply = async function (hookData) {
	const isInDevGroup = await Groups.isMember(hookData.data.uid, 'My developer group');
	if (isInDevGroup) {
		await topics.setTopicField(hookData.topic.tid, 'postedInByDeveloper', 1);
	}
};

patsanch

Nice! Thanks baris! I agree that's simpler