Search crashes MongoDB and NodeBB

julian

In that case, I'd definitely recommend Mongo be set up on a service, so it restarts if OOM killed...

Dave Jensen

@julian MongoDB was already a service but the setting to always restart had to be set, which I did sorry after the crash. It's not a default setting.
I'm thinking the setup docs need to get updated?

Dave Jensen

Two more things that I did that seemed to help:

Moved the data files to and XFS Volume. DigitalOcean now has XFS for their attached Volumes product so this was super easy.
- I should have been using mounted volumes to begin with just to protect the data.
The MongoDB logs made another recommendation to turn off transparent huge pages so I did it:
https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/

The site still occasionally runs slow but no crashes. I think removing the unnecessary topics will make the site blazing fast ... or at least no slow downs.

julian

Good idea re: THP. I don't really know how that changes things, but if it makes Mongo more performant (or reduces memory consumption on lower-memory servers), then that's great!

Dave Jensen

However, database workloads often perform poorly with THP, because they tend to have sparse rather than contiguous memory access patterns. You should disable THP on Linux machines to ensure best performance with MongoDB.

That's why.

Dave Jensen

So, after I turned off THP and put the data on an XFS volume. Search got better.

After I upgraded to 1.11.0, excluded certain categories, and re-indexed it got worse. When a search happens, it locks up MongoDB which crashes NodeBB. It also appears that search has a global read lock on the database.

If you all want to test changes on search in the future, I can give you a dump of my DB. I really don't like my forums crashing all the time, unreliable sites drive both Google and user traffic down.

Dave Jensen

@julianlam @baris @PitaJ

Out of curiosity, how much RAM does your MongoDB server(s) have?

It looks like you have 67k posts, right? In terms of what I want indexed, I have 5x that.

Looks like I might have to price out AWS CloudSearch vs. more RAM. ...

$67/month for a CloudSearch medium instance
8GB DigitalOcean instance is $40/month
16GB DigitalOcean instance is $80/month

<baris>

This forum is hosted on a 4gb($20/month) droplet. Can you post the raw output you get under MONGODB RAW INFO at /admin/advanced/database

Dave Jensen

{
    "db": "nodebb",
    "collections": 6,
    "views": 0,
    "objects": 13562976,
    "avgObjSize": 1406.2239958988353,
    "dataSize": 19072582307,
    "storageSize": 5140107264,
    "numExtents": 0,
    "indexes": 12,
    "indexSize": 2895679488,
    "fsUsedSize": 8594370560,
    "fsTotalSize": 21464350720,
    "ok": 1,
    "mem": {
        "bits": 64,
        "resident": "1.310",
        "virtual": "2.309",
        "supported": true,
        "mapped": "0.000",
        "mappedWithJournal": 0
    },
    "collectionData": [
        {
            "name": "nodebb.sessions",
            "count": 802700,
            "size": 205618438,
            "avgObjSize": 256,
            "storageSize": 104878080,
            "totalIndexSize": 52961280,
            "indexSizes": {
                "_id_": 44048384,
                "expires_1": 8912896
            }
        },
        {
            "name": "nodebb.socket.io",
            "count": 7166,
            "size": 5242523,
            "avgObjSize": 731,
            "storageSize": 1638400,
            "totalIndexSize": 151552,
            "indexSizes": {
                "_id_": 151552
            }
        },
        {
            "name": "nodebb.searchpost",
            "count": 1239083,
            "size": 2202184242,
            "avgObjSize": 1777,
            "storageSize": 835080192,
            "totalIndexSize": 2357145600,
            "indexSizes": {
                "_id_": 16314368,
                "content_text_uid_1_cid_1": 2340831232
            }
        },
        {
            "name": "nodebb.objects",
            "count": 11481491,
            "size": 16656741626,
            "avgObjSize": 1450,
            "storageSize": 4196724736,
            "totalIndexSize": 482164736,
            "indexSizes": {
                "_id_": 118857728,
                "_key_1_score_-1": 155598848,
                "_key_1_value_-1": 166690816,
                "expireAt_1": 41017344
            }
        },
        {
            "name": "nodebb.pubsub",
            "count": 1,
            "size": 30,
            "avgObjSize": 30,
            "storageSize": 16384,
            "totalIndexSize": 16384,
            "indexSizes": {
                "_id_": 16384
            }
        },
        {
            "name": "nodebb.searchtopic",
            "count": 32535,
            "size": 2795448,
            "avgObjSize": 85,
            "storageSize": 1769472,
            "totalIndexSize": 3239936,
            "indexSizes": {
                "_id_": 372736,
                "content_text_uid_1_cid_1": 2867200
            }
        }
    ],
    "network": {
        "bytesIn": 245956294,
        "bytesOut": 686747360,
        "physicalBytesIn": 245956294,
        "physicalBytesOut": 686747360,
        "numRequests": 760690,
        "compression": {
            "snappy": {
                "compressor": {
                    "bytesIn": 0,
                    "bytesOut": 0
                },
                "decompressor": {
                    "bytesIn": 0,
                    "bytesOut": 0
                }
            }
        },
        "serviceExecutorTaskStats": {
            "executor": "passthrough",
            "threadsRunning": 25
        }
    }
}

<baris>

@djensen47 said in Search crashes MongoDB and NodeBB:

{
"name": "nodebb.searchpost",
"count": 1239083,
"size": 2202184242,
"avgObjSize": 1777,
"storageSize": 835080192,
"totalIndexSize": 2357145600,
"indexSizes": {
"id": 16314368,
"content_text_uid_1_cid_1": 2340831232
}
},

Seems like still have 1.2m posts indexed

Dave Jensen

@baris Sorry I missed that bit. Should I drop those collections and try to re-index?

<baris>

If you click clear index on the ACP page for the plugin and then reindex it should only reindex the category not excluded.