Help Fine Tuning Advanced Cache Settings

patsanch

@baris Thanks for the explanation.

One thing I noticed though is that my Group Cache hit ratio is really low 0.15. What does that mean? Everything else is high.

<baris>

Can you post your stats? How many categories/groups do you have?

The group cache is used to store the membership of users in groups(and privileges). For example checking if a user is an administrator is done by checking if they are member of the administrators group. The results are cached since group membership rarely changes.
If the hit ratio is low you might have lots of active registered users and/or only a single nodebb instance. Each nodebb process has it's own cache and if users are distributed to these nodebb instances evenly the group cache will have a higher hit ratio.

patsanch

Sure, here you go.

Screen Shot 2021-02-24 at 10.02.12 AM.png

Right now, we have 12 active categories, 10 groups.

patsanch

I toggled the radio buttons. I guess that deleted the cache. I haven't done it for Group Cache.

patsanch

or maybe I hit trash when I was tinkering with it.

<baris>

The toggle lets you enable/disable the cache, the delete button just clears it's contents.

You can download the contents of the cache as well.

When you clear the group cache how long does it take to make that many misses?

How many users are on your nodebb? You can check /admin/development/info for more detailed stats.

My guess is there is some code that is making a lot of group membership checks maybe a plugin. Or you just have a lot of active users.

<baris>

You can download the contents of the group cache and examine it to get more info about what is being cached as well.

patsanch

@baris I deleted the Group Cache and it's full after 2 secs.

<baris>

@baris said in Help Fine Tuning Advanced Cache Settings:

/admin/development/info

Did you check this page? Share that one as well.

patsanch

@baris here's our development/info stats.

-deleted image link-

The image dimensions were too big to upload.

patsanch

ip-10-111-250-67:4567	 / 	7985	v10.17.0	82 / 4 / 93	v1.16.x@d7044a	1244.30 / 39.95	292.63 mb / 10635.03 mb	1.05, 1.64, 1.87	57m
ip-10-111-250-67:4568	 / 	7986	v10.17.0	89 / 7 / 106	v1.16.x@d7044a	1176.16 / 38.71	340.14 mb / 10650.05 mb	1.05, 1.64, 1.87	57m
ip-10-111-250-67:4569	 / 	7988	v10.17.0	106 / 12 / 150	v1.16.x@d7044a	1395.36 / 43.34	369.73 mb / 10650.05 mb	1.05, 1.64, 1.87	57m
ip-10-111-250-67:4570	 / 	8005	v10.17.0	97 / 5 / 123	v1.16.x@d7044a	1431.64 / 45.03	312.77 mb / 10650.05 mb	1.05, 1.64, 1.87	57m

[
    {
        "process": {
            "port": "4567",
            "pid": 7985,
            "title": "/usr/bin/node",
            "version": "v10.17.0",
            "memoryUsage": {
                "rss": 331522048,
                "heapTotal": 270622720,
                "heapUsed": 191571224,
                "external": 23467064,
                "humanReadable": "316.16"
            },
            "uptime": 3516.246,
            "cpuUsage": {
                "user": "1258.65",
                "system": "40.50"
            },
            "uptimeHumanReadable": "58m"
        },
        "os": {
            "hostname": "ip-10-111-250-67",
            "type": "Linux",
            "platform": "linux",
            "arch": "x64",
            "release": "5.4.0-1029-aws",
            "load": "1.70, 1.72, 1.89",
            "freemem": "10625.66",
            "totalmem": "16481.63"
        },
        "nodebb": {
            "isCluster": true,
            "isPrimary": true,
            "runJobs": true,
            "jobsDisabled": false
        },
        "git": {
            "hash": "d7044a5e8623ae6b95360ff58e8f6dbc0e90629b",
            "hashShort": "d7044a",
            "branch": "v1.16.x"
        },
        "stats": {
            "onlineGuestCount": 4,
            "onlineRegisteredCount": 82,
            "socketCount": 93,
            "users": {
                "categories": 0,
                "recent": 1,
                "unread": 0,
                "topics": 39,
                "category": 41
            },
            "topics": [
                {
                    "tid": "26200",
                    "count": 5
                },
                {
                    "tid": "26202",
                    "count": 4
                },
                {
                    "tid": "26203",
                    "count": 3
                },
                {
                    "tid": "26747",
                    "count": 3
                },
                {
                    "tid": "26359",
                    "count": 2
                },
                {
                    "tid": "26199",
                    "count": 2
                },
                {
                    "tid": "26211",
                    "count": 2
                },
                {
                    "tid": "26901",
                    "count": 2
                },
                {
                    "tid": "26634",
                    "count": 1
                },
                {
                    "tid": "25866",
                    "count": 1
                }
            ]
        },
        "id": "ip-10-111-250-67:4567"
    },
    {
        "process": {
            "port": "4568",
            "pid": 7986,
            "title": "/usr/bin/node",
            "version": "v10.17.0",
            "memoryUsage": {
                "rss": 370384896,
                "heapTotal": 302641152,
                "heapUsed": 261548072,
                "external": 22358009,
                "humanReadable": "353.23"
            },
            "uptime": 3516.239,
            "cpuUsage": {
                "user": "1204.29",
                "system": "39.61"
            },
            "uptimeHumanReadable": "58m"
        },
        "os": {
            "hostname": "ip-10-111-250-67",
            "type": "Linux",
            "platform": "linux",
            "arch": "x64",
            "release": "5.4.0-1029-aws",
            "load": "1.70, 1.72, 1.89",
            "freemem": "10625.66",
            "totalmem": "16481.63"
        },
        "nodebb": {
            "isCluster": true,
            "isPrimary": false,
            "runJobs": false,
            "jobsDisabled": false
        },
        "git": {
            "hash": "d7044a5e8623ae6b95360ff58e8f6dbc0e90629b",
            "hashShort": "d7044a",
            "branch": "v1.16.x"
        },
        "stats": {
            "onlineGuestCount": 5,
            "onlineRegisteredCount": 93,
            "socketCount": 110,
            "users": {
                "categories": 2,
                "recent": 2,
                "unread": 0,
                "topics": 41,
                "category": 48
            },
            "topics": [
                {
                    "tid": "26747",
                    "count": 7
                },
                {
                    "tid": "26201",
                    "count": 6
                },
                {
                    "tid": "26200",
                    "count": 3
                },
                {
                    "tid": "26202",
                    "count": 2
                },
                {
                    "tid": "26922",
                    "count": 1
                },
                {
                    "tid": "26359",
                    "count": 1
                },
                {
                    "tid": "26794",
                    "count": 1
                },
                {
                    "tid": "25999",
                    "count": 1
                },
                {
                    "tid": "26210",
                    "count": 1
                },
                {
                    "tid": "26136",
                    "count": 1
                }
            ]
        },
        "id": "ip-10-111-250-67:4568"
    },
    {
        "process": {
            "port": "4569",
            "pid": 7988,
            "title": "/usr/bin/node",
            "version": "v10.17.0",
            "memoryUsage": {
                "rss": 365465600,
                "heapTotal": 299364352,
                "heapUsed": 262696736,
                "external": 23254567,
                "humanReadable": "348.54"
            },
            "uptime": 3516.235,
            "cpuUsage": {
                "user": "1411.07",
                "system": "43.92"
            },
            "uptimeHumanReadable": "58m"
        },
        "os": {
            "hostname": "ip-10-111-250-67",
            "type": "Linux",
            "platform": "linux",
            "arch": "x64",
            "release": "5.4.0-1029-aws",
            "load": "1.70, 1.72, 1.89",
            "freemem": "10625.66",
            "totalmem": "16481.63"
        },
        "nodebb": {
            "isCluster": true,
            "isPrimary": false,
            "runJobs": false,
            "jobsDisabled": false
        },
        "git": {
            "hash": "d7044a5e8623ae6b95360ff58e8f6dbc0e90629b",
            "hashShort": "d7044a",
            "branch": "v1.16.x"
        },
        "stats": {
            "onlineGuestCount": 12,
            "onlineRegisteredCount": 105,
            "socketCount": 151,
            "users": {
                "categories": 2,
                "recent": 4,
                "unread": 0,
                "topics": 69,
                "category": 48
            },
            "topics": [
                {
                    "tid": "26200",
                    "count": 8
                },
                {
                    "tid": "26747",
                    "count": 8
                },
                {
                    "tid": "26201",
                    "count": 3
                },
                {
                    "tid": "26656",
                    "count": 3
                },
                {
                    "tid": "26927",
                    "count": 3
                },
                {
                    "tid": "26533",
                    "count": 2
                },
                {
                    "tid": "26209",
                    "count": 2
                },
                {
                    "tid": "26199",
                    "count": 2
                },
                {
                    "tid": "26727",
                    "count": 2
                },
                {
                    "tid": "26901",
                    "count": 2
                }
            ]
        },
        "id": "ip-10-111-250-67:4569"
    },
    {
        "process": {
            "port": "4570",
            "pid": 8005,
            "title": "/usr/bin/node",
            "version": "v10.17.0",
            "memoryUsage": {
                "rss": 336445440,
                "heapTotal": 272719872,
                "heapUsed": 218833992,
                "external": 22805176,
                "humanReadable": "320.86"
            },
            "uptime": 3516.232,
            "cpuUsage": {
                "user": "1454.85",
                "system": "45.98"
            },
            "uptimeHumanReadable": "58m"
        },
        "os": {
            "hostname": "ip-10-111-250-67",
            "type": "Linux",
            "platform": "linux",
            "arch": "x64",
            "release": "5.4.0-1029-aws",
            "load": "1.70, 1.72, 1.89",
            "freemem": "10616.37",
            "totalmem": "16481.63"
        },
        "nodebb": {
            "isCluster": true,
            "isPrimary": false,
            "runJobs": false,
            "jobsDisabled": false
        },
        "git": {
            "hash": "d7044a5e8623ae6b95360ff58e8f6dbc0e90629b",
            "hashShort": "d7044a",
            "branch": "v1.16.x"
        },
        "stats": {
            "onlineGuestCount": 5,
            "onlineRegisteredCount": 101,
            "socketCount": 128,
            "users": {
                "categories": 1,
                "recent": 2,
                "unread": 0,
                "topics": 63,
                "category": 46
            },
            "topics": [
                {
                    "tid": "26200",
                    "count": 10
                },
                {
                    "tid": "26747",
                    "count": 6
                },
                {
                    "tid": "26201",
                    "count": 6
                },
                {
                    "tid": "26199",
                    "count": 5
                },
                {
                    "tid": "26359",
                    "count": 4
                },
                {
                    "tid": "26203",
                    "count": 4
                },
                {
                    "tid": "26210",
                    "count": 3
                },
                {
                    "tid": "26728",
                    "count": 2
                },
                {
                    "tid": "26656",
                    "count": 2
                },
                {
                    "tid": "26204",
                    "count": 2
                }
            ]
        },
        "id": "ip-10-111-250-67:4570"
    }
]

<baris>

Your stats look fine, you have around ~100-150 users per nodebb instance.

Do you have custom plugins? I suggest clearing the group cache and then downloading it after it is full in 2 seconds. Then you can check what is being cached.

patsanch

@baris no custom plugins.

Screen Shot 2021-02-24 at 10.22.19 AM.png

Okay, i'll do what you suggested.

patsanch

@baris Can I increase group cache size? just curious.

I'm reviewing what's on the cache dump. Looks like lots of permission stuff. We have multiple moderators per category. Maybe that's adding up to that list. I can probably clean that up and see how it looks after.

<baris>

Sure you can change it in code, https://github.com/NodeBB/NodeBB/blob/master/src/groups/cache.js#L8.

Modify the 40000 and restart all instances.

patsanch

@baris Thanks! I'll try to cleanup our privileges first. If that doesn't help, I'll tweak the value in code.

gotwf

@baris said in Help Fine Tuning Advanced Cache Settings:

@baris said in Help Fine Tuning Advanced Cache Settings:

/admin/development/info

New one for me. And pretty nice. Is this documented somewhere? Or is it one of those top seekrit dev thangs...

Thx--

patsanch

@baris Okay, I did 1M and it's almost full. Hit ratio is a nice 0.9397.

Thanks!

<baris>

Make sure to check your memory usage on the other page. Not sure how much it will increase it with a million entries.

<baris>

@gotwf yeah it's meant for devs