Search crashes MongoDB and NodeBB

Technical Support
  • I re-indexed 1.2M posts and now my site can't stay up .

    Basically MongoDB runs out of memory, goes down, and then NodeBB crashes because it cannot connect.

    1. What's the solution here? THis can't be the first time. I already upped the memory on the instance but unless I go for something super expensive I doubt it will solve the issue.
    2. Can I re-index only specific categories?
    3. Why is NodeBB so fragile when it comes to connecting to the DB? A few connection failures and crash? Really? This seems to be the case with any error the system encounters. C an I use PM2 or forever with NodeBB?
  • Also, DB Search is super slow. Does it even use any MongoDB indexes?

    (Sorry if I seem a little cranky, noisy server alerts kept me up all night)

  • What are the specs of your server? nodebb-plugin-dbsearch uses indexes so search shouldn't be slow.

  • Digital Ocean 3GB 1 vCPU instance

  • I'm still also worried about the resilience of the NodeBB server. Why does it just crash and not try to restart? This is especially worrisome given that it's also recommended not to use PM2.

  • @djensen47 if you run it with ./nodebb start I think it should restart automatically

  • @PitaJ It doesn't seem like it. I think I need to set up syatemctrl [sic]

  • @djensen47 I have a systemd config usable here:

    It should automatically kick NodeBB back up if it goes down.

    That said, NodeBB shouldn't crash, if it is, you should inspect the logs and take a look as to why...

  • It was crashing because it lost the connection to MongoDB. MongoDB was crashing because DB Search would cause MongoDB to run out of memory. I tried up to a 4GB instance and it would still crash OOM on some searches.

    I didn't have systemd set up. For some reason I thought NodeBB had a watchdog like forever watching it so I used forever. The instructions for systemd seemed to have specific requirements in the docs that were a little confusing so I went with the forever option.

  • @djensen47 I'm pay sure that ./nodebb start does have something like that. But if the database is crashed, it'll only try to start up NodeBB so many times before giving up.

  • In that case, I'd definitely recommend Mongo be set up on a service, so it restarts if OOM killed...

  • @julian MongoDB was already a service but the setting to always restart had to be set, which I did sorry after the crash. It's not a default setting.
    I'm thinking the setup docs need to get updated?

  • Two more things that I did that seemed to help:

    • Moved the data files to and XFS Volume. DigitalOcean now has XFS for their attached Volumes product so this was super easy.
      • I should have been using mounted volumes to begin with just to protect the data.
    • The MongoDB logs made another recommendation to turn off transparent huge pages so I did it:

    The site still occasionally runs slow but no crashes. I think removing the unnecessary topics will make the site blazing fast ... or at least no slow downs.

  • Good idea re: THP. I don't really know how that changes things, but if it makes Mongo more performant (or reduces memory consumption on lower-memory servers), then that's great!

  • However, database workloads often perform poorly with THP, because they tend to have sparse rather than contiguous memory access patterns. You should disable THP on Linux machines to ensure best performance with MongoDB.

    That's why. 😉

  • So, after I turned off THP and put the data on an XFS volume. Search got better.

    After I upgraded to 1.11.0, excluded certain categories, and re-indexed it got worse. When a search happens, it locks up MongoDB which crashes NodeBB. It also appears that search has a global read lock on the database.

    If you all want to test changes on search in the future, I can give you a dump of my DB. I really don't like my forums crashing all the time, unreliable sites drive both Google and user traffic down.

  • @julianlam @baris @PitaJ

    Out of curiosity, how much RAM does your MongoDB server(s) have?

    It looks like you have 67k posts, right? In terms of what I want indexed, I have 5x that.

    Looks like I might have to price out AWS CloudSearch vs. more RAM. 😞 ...

    $67/month for a CloudSearch medium instance
    8GB DigitalOcean instance is $40/month
    16GB DigitalOcean instance is $80/month

  • This forum is hosted on a 4gb($20/month) droplet. Can you post the raw output you get under MONGODB RAW INFO at /admin/advanced/database

  • {
        "db": "nodebb",
        "collections": 6,
        "views": 0,
        "objects": 13562976,
        "avgObjSize": 1406.2239958988353,
        "dataSize": 19072582307,
        "storageSize": 5140107264,
        "numExtents": 0,
        "indexes": 12,
        "indexSize": 2895679488,
        "fsUsedSize": 8594370560,
        "fsTotalSize": 21464350720,
        "ok": 1,
        "mem": {
            "bits": 64,
            "resident": "1.310",
            "virtual": "2.309",
            "supported": true,
            "mapped": "0.000",
            "mappedWithJournal": 0
        "collectionData": [
                "name": "nodebb.sessions",
                "count": 802700,
                "size": 205618438,
                "avgObjSize": 256,
                "storageSize": 104878080,
                "totalIndexSize": 52961280,
                "indexSizes": {
                    "_id_": 44048384,
                    "expires_1": 8912896
                "name": "",
                "count": 7166,
                "size": 5242523,
                "avgObjSize": 731,
                "storageSize": 1638400,
                "totalIndexSize": 151552,
                "indexSizes": {
                    "_id_": 151552
                "name": "nodebb.searchpost",
                "count": 1239083,
                "size": 2202184242,
                "avgObjSize": 1777,
                "storageSize": 835080192,
                "totalIndexSize": 2357145600,
                "indexSizes": {
                    "_id_": 16314368,
                    "content_text_uid_1_cid_1": 2340831232
                "name": "nodebb.objects",
                "count": 11481491,
                "size": 16656741626,
                "avgObjSize": 1450,
                "storageSize": 4196724736,
                "totalIndexSize": 482164736,
                "indexSizes": {
                    "_id_": 118857728,
                    "_key_1_score_-1": 155598848,
                    "_key_1_value_-1": 166690816,
                    "expireAt_1": 41017344
                "name": "nodebb.pubsub",
                "count": 1,
                "size": 30,
                "avgObjSize": 30,
                "storageSize": 16384,
                "totalIndexSize": 16384,
                "indexSizes": {
                    "_id_": 16384
                "name": "nodebb.searchtopic",
                "count": 32535,
                "size": 2795448,
                "avgObjSize": 85,
                "storageSize": 1769472,
                "totalIndexSize": 3239936,
                "indexSizes": {
                    "_id_": 372736,
                    "content_text_uid_1_cid_1": 2867200
        "network": {
            "bytesIn": 245956294,
            "bytesOut": 686747360,
            "physicalBytesIn": 245956294,
            "physicalBytesOut": 686747360,
            "numRequests": 760690,
            "compression": {
                "snappy": {
                    "compressor": {
                        "bytesIn": 0,
                        "bytesOut": 0
                    "decompressor": {
                        "bytesIn": 0,
                        "bytesOut": 0
            "serviceExecutorTaskStats": {
                "executor": "passthrough",
                "threadsRunning": 25
  • @djensen47 said in Search crashes MongoDB and NodeBB:

    "name": "nodebb.searchpost",
    "count": 1239083,
    "size": 2202184242,
    "avgObjSize": 1777,
    "storageSize": 835080192,
    "totalIndexSize": 2357145600,
    "indexSizes": {
    "id": 16314368,
    "content_text_uid_1_cid_1": 2340831232

    Seems like still have 1.2m posts indexed

Suggested Topics

  • 0 Votes
    11 Posts

    @pitaj Well, this info is golden.

    This clearly needs to be documented well and easily found. The usage message of ./nodebb setup --help should indicate the expected syntax and strikingly point out config.json is the wrong syntax in this case to be fed into ./nodebb setup.

    In the long run, there should be no special case or different syntax between node app --setup and ./nodebb setup.

    Thank you, @PitaJ, @Max-0 .

  • 1 Votes
    1 Posts
    NodeBB version: 1.16.2 NodeBB git hash: 023e4cfcee208a0bae121a6e343a8c53009fd090 NodeJS version: v15.11.0. Installed NodeBB Plugins: No Database type: mongo Database version: v4.4.4 Exact steps to cause this issue: "mongo": { "host": "", "port": "27017", "username": "nodebb", "password": "nodebb", "database": "nodebb", "url": "mongodb://nodebb:#[email protected]:27017/nodebb?ssl=true&authSource=admin" },

    node app --setup

    What happened instead: Now configuring mongo database: 2021-03-12T17:06:56.884Z [18868] - warn: NodeBB Setup Aborted. MongoServerSelectionError: connection <monitor> to closed at Timeout._onTimeout (C:\Codes\NodeBB\node_modules\mongodb\lib\core\sdam\topology.js:438:30) at listOnTimeout (node:internal/timers:557:17) at processTimers (node:internal/timers:500:7)
  • 0 Votes
    9 Posts

    You've put port 8080 in "url". This should be the actual URL that people visit, and not the port on which the server should listen. You can specify the actual listen-port in config.json as follows:

    "port": 8080,

    This will fix the emoji-issue you're talking about 😉 (along with many other problems that may come up because the clients are trying an unresolvable port 8080)


  • 0 Votes
    3 Posts


    I also think it was MongoDB as well because I noticed that the database was set to "0" (default MongoDB on NodeBB setup) and when I changed it to the nodebb database it was as if it was default. Very frustrating but considering the forum was relatively new, not much was lost.