Questions about the setup for community.nodebb.org

[djensen47]

@baris said in Search crashes MongoDB and NodeBB:

This forum is hosted on a 4gb($20/month) droplet.

That's 4GB, 2 CPUs, and 80GB SSD

1. Is the entire forum hosted on the same server? NodeBB and MongoDB?
2. Are you using clustering?
3. If yes to clustering, are you setup for two instances (or for each CPU)? Or more?
4. Are you using a Redis server for pub/sub for sessions?
5. Do you have 6000 pageview per hour from bots?

[djensen47]

This post is deleted!

[baris]

1. Is the entire forum hosted on the same server? NodeBB and MongoDB?

Yes

1. Are you using clustering?

Yes

1. If yes to clustering, are you setup for two instances (or for each CPU)? Or more?

2 node procs

1. Are you using a Redis server for pub/sub for sessions?

Yes redis is used for pubsub and sessions

1. Do you have 6000 pageview per hour from bots?

Pageviews from bots range from 2.5k to 5k per hour. And seesm to spike every 4 days.

[djensen47]

Thanks, 3 more ...

6. Do you manually set the amount of memory for WiredTiger or just go with the default 50% - 1GB?
7. Any special settings for redis?
8. Do you use nginx to serve static assets?

[djensen47]

(Was going to update the existing 3 to 4 but saw that you might already be answering those)

9. About how much memory is being used by
   • Each nodejs instance
   • Redis (answered 250MB)
   • MongoDB (though this should be the same as #6 above but I noticed my instance uses more)

[baris]

6. Do you manually set the amount of memory for WiredTiger or just go with the default 50% - 1GB?

Default settings

7. Any special settings for redis?

in redis.conf

maxmemory 250000000 
maxmemory-policy volatile-lru

This limits redis to 250mb memory and evicts least recently used keys when memory is reaches 250mb.

8. Do you use nginx to serve static assets?

Yes

[PitaJ]

IIRC nginx also handles the 503 page

[baris]

Node instances use around ~220mb it can go higher depending on traffic.

[djensen47]

@baris Damn, mine is using 750MB+ but I guess half of that is closer to your number. I might have more plugins, which might effect the size in memory?

[djensen47]

Thanks for the info. It looks like I need 2 CPUs for the nodebb server and to setup a redis instance (somewhere).

Last thing, do 503s caused by load get logged? If not, how can I change it so that they are logged. I want to measure current 503 occurrences vs. any after I make these changes. Basically, I want to make sure throwing more money at the problem (more/better instances) fixes it.

[faizanzahid]

do you know the correct way to add clustering on node bb ?

[djensen47]

@faizanzahid Yes, it's in the docs: https://docs.nodebb.org/configuring/scaling/

Do not create more processes than CPUs. It's not advised as the results of doing such a thing are unpredictable.

[djensen47]

Ugh, I turned on clustering and now Socket.IO is broken ...

POST https://www.axisandallies.org/forums/socket.io/?EIO=3&transport=polling&t=MUiNOWE&sid=q1ikZjbjopxQbJZxAAYo 400 (Bad Request)

[djensen47]

I had to set the load balancing algorithm in HAProxy to source which is the same or similar to ip_hash in nginx.

[Tom_Rade]

@djensen47 What did you set it to?

[djensen47]

    balance source
    hash-type consistent

I've updated my haproxy config here: https://community.nodebb.org/topic/13315/haproxy-config

It probably needs a little more explanation that is there but it's an example to get you started.

[djensen47]

@baris How do I know if the pub/sub is working for sessions?

Some uses are reporting the logout problem after I turned on clustering and setup redis for pub/sub. But I haven't experienced it yet today but I haven't been on much.

Until I turned on IP sticky sessions, the site was erroring out on websockets (which might be unrelated).

[julian]

@djensen47 said in Questions about the setup for community.nodebb.org:

Until I turned on IP sticky sessions, the site was erroring out on websockets (which might be unrelated).

Not unrelated -- socket.io's handshake requires two requests that need to land on the same server, so the sticky sessions is a must

[djensen47]

@julian Thanks for the info! Given that it went away when I setup sticky sessions, that makes sense

It looks like redis is getting session keys. Does 120k session keys seem a little high?

[baris]

I think bots that hit the site generate a session key as well, so it might be right.