Widget persists and does not respond to any updates.

[attis]

Hi,

We have a procedure to let users know about scheduled maintenance by adding a global widget with appropriate info:

After the upgrade is compete we have simply ticked "Hide from anonymous users" and "Hide from registered users". This effectively hid it for everyone. At least in the past. Now, after upgrading to 1.9.0 the widget persists no matter what I do. It does not respond to edits of the content (it is reflected in ACP but not on the Forums pages).

I even removed the widget completely:

It is still there, informing about past maintenance

How can I get rid of the damn thing? Any ideas what's wrong?

Thanks!

[EDIT]
Further intel: We have two instances behind a load balancer. It seems like the changes to widgets are not synchronized... By removing "session stickiness" cookie I was able to access the second instance and indeed the stubborn widget was still present there in ACP. Removing it helped. The widget is gone. Now the question is: why wasn't two instances synchronized? Aren't widgets stored in database? Isn't Redis pub-sub mechanism supposed to sync the all instances in any case?
Definitely something's off here...

[baris]

Yes the widgets are saved in the database, but there is an object cache in the mongodb module so each nodebb instance caches some data, if the data is changed then the cache is cleared on all nodebb procs with redis/pubsub. I will try to replicate this on a multi process instance.

[baris]

@attis I wasn't able to reproduce on a 2 instance forum, can you put console.log(key); here and try saving on the widgets page, it should show the cache keys that are deleted.

[attis]

Will try to work on this on Monday. We'll see if I'm still able to reproduce.
Thanks.

[attis]

Ok. Still being able to reproduce - this time I've added a new item to navigation bar and it did not show up for users that are directed to second instance. Restarting both instances (i.e. forcefully clearing local caches) did the trick. Definitely something's off here. I guess I need to play around with console.log tomorrow and try to figure out what broke down.

[attis]

@baris I've added some debug code in the place you mentioned. Having opened two browsers, each connected to a different NodeBB instance I see the following behaviour:

• there's a lot of keys being cleared all the time
• among them, if I modify widgets I see widgets:global being cleared but only on the instance that initiated the change.
• the second instance shows no sign of clearing widgets:global (and the change obviously is not visible on it)
• this is consistent for any other change - navbar changes, profile edits...

So the question is: should I not be seeing the same cache key being cleared on both instances?

[UPDATE]
This looks interesting:

PUBSUB CHANNELS
*2
$27
db:0:adapter_key-request#/#
$28
db:0:adapter_key-response#/#

I see no pubsub_channel in our Redis instance...

[UPDATE 2]
It all seems clear to me now. In https://github.com/NodeBB/NodeBB/commit/e85aabbe74d7838185d0b102bf37a20c1a1b62a1 a logic was added that looks at isCluster to determine whether initialize "real" pub/sub mechanism or a "fake" local only. In our scenario we have instances running on separate machines and each instance is running just one NodeBB process. Therefore isCluster is false as determined by process.env.isCluster = ports.length > 1; in loader.js:118.

I am fixing this temporarily in our codebase by reversing the if...else if...else..if so that is does not look at isCluster in src/pubsub.js because I am not sure if "faking" or "forcing" NodeBB to recognize our scenario as clustered is correct and would not have any side-effects.

Advices?

To clarify, the code after my fix looks like this:

	var pubsub;

	if (nconf.get('redis')) {
		pubsub = require('./database/redis/pubsub');
	} else if (nconf.get('mongo')) {
		pubsub = require('./database/mongo/pubsub');
	} else {
		var EventEmitter = require('events');
		pubsub = new EventEmitter();
		pubsub.publish = pubsub.emit.bind(pubsub);
	}

Feel free to borrow it if it looks okay to you. It works for us

[baris]

@attis Thanks for debugging, I added a new config setting into config.json, you can set isCluster: true in your config.json and it should work now. It will be available in 1.9.2 so you can keep your change until then. https://github.com/NodeBB/NodeBB/issues/6509