NodeBB + MongoDB + Scaling (k8s)

Antosik

Hey!

We are planning to launch a forum that will be powered by NodeBB.

We have 3 pods for MongoDB as well as 3 pods (with auto-scaling enabled) NodeBB.
It all works on Kubernetes + AWS ALB.

We recently faced a problem with scaling (working in a cluster).
When set in the forum options settings, they only apply to the current instance and not to all NodeBB instances.
The MongoDB database contains the correct parameter value.

I think this is due to the NodeBB cache, but I don't want to disable it (produces additional cost).
Can the isCluster parameter help with the problem?

PitaJ

Can you give an example of what options aren't applying?

The easiest way is to restart after making changes.

What's your config look like for each instance? What version of NodeBB are you on?

Antosik

@pitaj
NodeBB version - v1.16.2
For example, brand:logo. Some instances have it and some of them - no.

Config template:

{
    "url": "${NODEBB_URL}",
    "port": "${NODEBB_PORT}",
    "secret": "${NODEBB_SECRET}",
    "database": "mongo",
    "mongo": {
        "host": "${MONGO_HOST}",
        "port": "${MONGO_PORT}",
        "username": "${MONGO_USERNAME}",
        "password": "${MONGO_PASS}",
        "database": "${MONGO_DATABASE}"
    },
    "socket.io": {
        "transports": ["websocket"]
    },
    "jobsDisabled": true,
    "isCluster": false
}

<baris>

If you are using clustering then you need to setup redis so when you make a change in the ACP it gets sent to all other nodebb instances. Also you shouldn't put isCluster: false in config.json if you are using clustering.

Antosik

@baris
Is there any way to not use Redis (it produces extra cost)?
Maybe we can implement something to support isCluster: true with the MongoDB?

Also you shouldn't put isCluster: false in config.json if you are using clustering.

We did it cause this property anyway works only with the Redis

gotwf

@antosik Maybe switch the entire db backend over to PostgreSQL? I think it is managing key-value session store w/o needing Redis?

Antosik

@gotwf
Does it support the isCluster config property?

PitaJ

@baris mind explaining why redis is now required for clustering?

PitaJ

@antosik to answer your question:

No, you must use redis is you have multiple instances.

Why are you so opposed to using redis? The amount of "extra cost" it adds can't be that significant when you're already running multiple instances of NodeBB.

<baris>

The dependencies for pubsub/socket.io are not maintained for mongodb/psql.

Antosik

@PitaJ
I'm not against it ^^.
Just want to research a bit before giving AWS more money. :d

@baris
So, we need to try:

implement something like src\database\redis\pubsub.js in mongo;
check out socket.io-adapter-mongo and @nodebb/mubsub packages?

UPD. Found an interesting thing - maybe we can use watch + changeStream to implement the pubsub ?

oplik0

The main issue from what I understood is actually lack of a maintained MongoDB equivalent of this: https://github.com/socketio/socket.io-redis

There are a few old packages based on the official redis one, but they were last published a few years ago. I think NodeBB team was looking at this fork recently (2 PRs made by 2 NodeBB staff members): https://github.com/lklepner/socket.io-adapter-mongo - and I guess it is the most recent one - but it also seems kinda abandoned.

So for now it seems that unless NodeBB takes on maintaining an adapter, redis will have to remain a requirement.

Antosik

@oplik0
Thanks for the detailed answer!

Speaking of Redis - what data is stored there if the main base is Mongo?
Does it required only for pub/sub or we also store the session data in it?

gotwf

@baris Ah, so PostgreSQL is not actually fully supported back end. Thx. Good to know. It was next on my list of experiments.

<baris>

When you have both mongo and redis blocks in config.json with "database": "mongo". MongoDB will be used to store the forum data and redis will be used for sessions & pubsub.

scottalanmiller

@antosik said in NodeBB + MongoDB + Scaling (k8s):

@PitaJ
I'm not against it ^^.
Just want to research a bit before giving AWS more money. :d

@baris
So, we need to try:

implement something like src\database\redis\pubsub.js in mongo;

check out socket.io-adapter-mongo and @nodebb/mubsub packages?

UPD. Found an interesting thing - maybe we can use watch + changeStream to implement the pubsub ?

What's your overall goal? I have a feeling that you are giving AWS way more money than you should already. If the goal is to go big without wasting money, this isn't the approach. Unless you are doing something in the billion hits a month range, I don't think that this approach can be cost effective at all.

So if you want to give AWS money just for fun, then you might as well give more. If you want to be cost effective, then setting yourself up for scaling will require more scale than will ever be practically needed.