Search crashes MongoDB and NodeBB



  • I re-indexed 1.2M posts and now my site can't stay up .

    Basically MongoDB runs out of memory, goes down, and then NodeBB crashes because it cannot connect.

    1. What's the solution here? THis can't be the first time. I already upped the memory on the instance but unless I go for something super expensive I doubt it will solve the issue.
    2. Can I re-index only specific categories?
    3. Why is NodeBB so fragile when it comes to connecting to the DB? A few connection failures and crash? Really? This seems to be the case with any error the system encounters. C an I use PM2 or forever with NodeBB?


  • Also, DB Search is super slow. Does it even use any MongoDB indexes?

    (Sorry if I seem a little cranky, noisy server alerts kept me up all night)


  • Admin

    What are the specs of your server? nodebb-plugin-dbsearch uses indexes so search shouldn't be slow.



  • Digital Ocean 3GB 1 vCPU instance



  • I'm still also worried about the resilience of the NodeBB server. Why does it just crash and not try to restart? This is especially worrisome given that it's also recommended not to use PM2.


  • Global Moderator

    @djensen47 if you run it with ./nodebb start I think it should restart automatically



  • @PitaJ It doesn't seem like it. I think I need to set up syatemctrl [sic]


  • Admin

    @djensen47 I have a systemd config usable here: https://docs.nodebb.org/configuring/running/

    It should automatically kick NodeBB back up if it goes down.

    That said, NodeBB shouldn't crash, if it is, you should inspect the logs and take a look as to why...



  • It was crashing because it lost the connection to MongoDB. MongoDB was crashing because DB Search would cause MongoDB to run out of memory. I tried up to a 4GB instance and it would still crash OOM on some searches.

    I didn't have systemd set up. For some reason I thought NodeBB had a watchdog like forever watching it so I used forever. The instructions for systemd seemed to have specific requirements in the docs that were a little confusing so I went with the forever option.


  • Global Moderator

    @djensen47 I'm pay sure that ./nodebb start does have something like that. But if the database is crashed, it'll only try to start up NodeBB so many times before giving up.


  • Admin

    In that case, I'd definitely recommend Mongo be set up on a service, so it restarts if OOM killed...



  • @julian MongoDB was already a service but the setting to always restart had to be set, which I did sorry after the crash. It's not a default setting.
    I'm thinking the setup docs need to get updated?



  • Two more things that I did that seemed to help:

    • Moved the data files to and XFS Volume. DigitalOcean now has XFS for their attached Volumes product so this was super easy.
      • I should have been using mounted volumes to begin with just to protect the data.
    • The MongoDB logs made another recommendation to turn off transparent huge pages so I did it:
      https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/

    The site still occasionally runs slow but no crashes. I think removing the unnecessary topics will make the site blazing fast ... or at least no slow downs.


  • Admin

    Good idea re: THP. I don't really know how that changes things, but if it makes Mongo more performant (or reduces memory consumption on lower-memory servers), then that's great!



  • However, database workloads often perform poorly with THP, because they tend to have sparse rather than contiguous memory access patterns. You should disable THP on Linux machines to ensure best performance with MongoDB.

    That's why. 😉



  • So, after I turned off THP and put the data on an XFS volume. Search got better.

    After I upgraded to 1.11.0, excluded certain categories, and re-indexed it got worse. When a search happens, it locks up MongoDB which crashes NodeBB. It also appears that search has a global read lock on the database.

    If you all want to test changes on search in the future, I can give you a dump of my DB. I really don't like my forums crashing all the time, unreliable sites drive both Google and user traffic down.



  • @julianlam @baris @PitaJ

    Out of curiosity, how much RAM does your MongoDB server(s) have?

    It looks like you have 67k posts, right? In terms of what I want indexed, I have 5x that.

    Looks like I might have to price out AWS CloudSearch vs. more RAM. 😞 ...


    $67/month for a CloudSearch medium instance
    8GB DigitalOcean instance is $40/month
    16GB DigitalOcean instance is $80/month


  • Admin

    This forum is hosted on a 4gb($20/month) droplet. Can you post the raw output you get under MONGODB RAW INFO at /admin/advanced/database



  • {
        "db": "nodebb",
        "collections": 6,
        "views": 0,
        "objects": 13562976,
        "avgObjSize": 1406.2239958988353,
        "dataSize": 19072582307,
        "storageSize": 5140107264,
        "numExtents": 0,
        "indexes": 12,
        "indexSize": 2895679488,
        "fsUsedSize": 8594370560,
        "fsTotalSize": 21464350720,
        "ok": 1,
        "mem": {
            "bits": 64,
            "resident": "1.310",
            "virtual": "2.309",
            "supported": true,
            "mapped": "0.000",
            "mappedWithJournal": 0
        },
        "collectionData": [
            {
                "name": "nodebb.sessions",
                "count": 802700,
                "size": 205618438,
                "avgObjSize": 256,
                "storageSize": 104878080,
                "totalIndexSize": 52961280,
                "indexSizes": {
                    "_id_": 44048384,
                    "expires_1": 8912896
                }
            },
            {
                "name": "nodebb.socket.io",
                "count": 7166,
                "size": 5242523,
                "avgObjSize": 731,
                "storageSize": 1638400,
                "totalIndexSize": 151552,
                "indexSizes": {
                    "_id_": 151552
                }
            },
            {
                "name": "nodebb.searchpost",
                "count": 1239083,
                "size": 2202184242,
                "avgObjSize": 1777,
                "storageSize": 835080192,
                "totalIndexSize": 2357145600,
                "indexSizes": {
                    "_id_": 16314368,
                    "content_text_uid_1_cid_1": 2340831232
                }
            },
            {
                "name": "nodebb.objects",
                "count": 11481491,
                "size": 16656741626,
                "avgObjSize": 1450,
                "storageSize": 4196724736,
                "totalIndexSize": 482164736,
                "indexSizes": {
                    "_id_": 118857728,
                    "_key_1_score_-1": 155598848,
                    "_key_1_value_-1": 166690816,
                    "expireAt_1": 41017344
                }
            },
            {
                "name": "nodebb.pubsub",
                "count": 1,
                "size": 30,
                "avgObjSize": 30,
                "storageSize": 16384,
                "totalIndexSize": 16384,
                "indexSizes": {
                    "_id_": 16384
                }
            },
            {
                "name": "nodebb.searchtopic",
                "count": 32535,
                "size": 2795448,
                "avgObjSize": 85,
                "storageSize": 1769472,
                "totalIndexSize": 3239936,
                "indexSizes": {
                    "_id_": 372736,
                    "content_text_uid_1_cid_1": 2867200
                }
            }
        ],
        "network": {
            "bytesIn": 245956294,
            "bytesOut": 686747360,
            "physicalBytesIn": 245956294,
            "physicalBytesOut": 686747360,
            "numRequests": 760690,
            "compression": {
                "snappy": {
                    "compressor": {
                        "bytesIn": 0,
                        "bytesOut": 0
                    },
                    "decompressor": {
                        "bytesIn": 0,
                        "bytesOut": 0
                    }
                }
            },
            "serviceExecutorTaskStats": {
                "executor": "passthrough",
                "threadsRunning": 25
            }
        }
    }
    

  • Admin

    @djensen47 said in Search crashes MongoDB and NodeBB:

    {
    "name": "nodebb.searchpost",
    "count": 1239083,
    "size": 2202184242,
    "avgObjSize": 1777,
    "storageSize": 835080192,
    "totalIndexSize": 2357145600,
    "indexSizes": {
    "id": 16314368,
    "content_text_uid_1_cid_1": 2340831232
    }
    },

    Seems like still have 1.2m posts indexed


 

Suggested Topics

| |