Search crashes MongoDB and NodeBB
-
Also, DB Search is super slow. Does it even use any MongoDB indexes?
(Sorry if I seem a little cranky, noisy server alerts kept me up all night)
-
Digital Ocean 3GB 1 vCPU instance
-
I'm still also worried about the resilience of the NodeBB server. Why does it just crash and not try to restart? This is especially worrisome given that it's also recommended not to use PM2.
-
@djensen47 if you run it with
./nodebb startI think it should restart automatically -
@djensen47 I have a systemd config usable here: https://docs.nodebb.org/configuring/running/
It should automatically kick NodeBB back up if it goes down.
That said, NodeBB shouldn't crash, if it is, you should inspect the logs and take a look as to why...
-
It was crashing because it lost the connection to MongoDB. MongoDB was crashing because DB Search would cause MongoDB to run out of memory. I tried up to a 4GB instance and it would still crash OOM on some searches.
I didn't have systemd set up. For some reason I thought NodeBB had a watchdog like forever watching it so I used forever. The instructions for systemd seemed to have specific requirements in the docs that were a little confusing so I went with the forever option.
-
@djensen47 I'm pay sure that
./nodebb startdoes have something like that. But if the database is crashed, it'll only try to start up NodeBB so many times before giving up. -
Two more things that I did that seemed to help:
- Moved the data files to and XFS Volume. DigitalOcean now has XFS for their attached Volumes product so this was super easy.
- I should have been using mounted volumes to begin with just to protect the data.
- The MongoDB logs made another recommendation to turn off transparent huge pages so I did it:
https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/
The site still occasionally runs slow but no crashes. I think removing the unnecessary topics will make the site blazing fast ... or at least no slow downs.
- Moved the data files to and XFS Volume. DigitalOcean now has XFS for their attached Volumes product so this was super easy.
-
However, database workloads often perform poorly with THP, because they tend to have sparse rather than contiguous memory access patterns. You should disable THP on Linux machines to ensure best performance with MongoDB.
That's why.
-
So, after I turned off THP and put the data on an XFS volume. Search got better.
After I upgraded to 1.11.0, excluded certain categories, and re-indexed it got worse. When a search happens, it locks up MongoDB which crashes NodeBB. It also appears that search has a global read lock on the database.
If you all want to test changes on search in the future, I can give you a dump of my DB. I really don't like my forums crashing all the time, unreliable sites drive both Google and user traffic down.
-
Out of curiosity, how much RAM does your MongoDB server(s) have?
It looks like you have 67k posts, right? In terms of what I want indexed, I have 5x that.
Looks like I might have to price out AWS CloudSearch vs. more RAM.
...
$67/month for a CloudSearch medium instance
8GB DigitalOcean instance is $40/month
16GB DigitalOcean instance is $80/month -
{ "db": "nodebb", "collections": 6, "views": 0, "objects": 13562976, "avgObjSize": 1406.2239958988353, "dataSize": 19072582307, "storageSize": 5140107264, "numExtents": 0, "indexes": 12, "indexSize": 2895679488, "fsUsedSize": 8594370560, "fsTotalSize": 21464350720, "ok": 1, "mem": { "bits": 64, "resident": "1.310", "virtual": "2.309", "supported": true, "mapped": "0.000", "mappedWithJournal": 0 }, "collectionData": [ { "name": "nodebb.sessions", "count": 802700, "size": 205618438, "avgObjSize": 256, "storageSize": 104878080, "totalIndexSize": 52961280, "indexSizes": { "_id_": 44048384, "expires_1": 8912896 } }, { "name": "nodebb.socket.io", "count": 7166, "size": 5242523, "avgObjSize": 731, "storageSize": 1638400, "totalIndexSize": 151552, "indexSizes": { "_id_": 151552 } }, { "name": "nodebb.searchpost", "count": 1239083, "size": 2202184242, "avgObjSize": 1777, "storageSize": 835080192, "totalIndexSize": 2357145600, "indexSizes": { "_id_": 16314368, "content_text_uid_1_cid_1": 2340831232 } }, { "name": "nodebb.objects", "count": 11481491, "size": 16656741626, "avgObjSize": 1450, "storageSize": 4196724736, "totalIndexSize": 482164736, "indexSizes": { "_id_": 118857728, "_key_1_score_-1": 155598848, "_key_1_value_-1": 166690816, "expireAt_1": 41017344 } }, { "name": "nodebb.pubsub", "count": 1, "size": 30, "avgObjSize": 30, "storageSize": 16384, "totalIndexSize": 16384, "indexSizes": { "_id_": 16384 } }, { "name": "nodebb.searchtopic", "count": 32535, "size": 2795448, "avgObjSize": 85, "storageSize": 1769472, "totalIndexSize": 3239936, "indexSizes": { "_id_": 372736, "content_text_uid_1_cid_1": 2867200 } } ], "network": { "bytesIn": 245956294, "bytesOut": 686747360, "physicalBytesIn": 245956294, "physicalBytesOut": 686747360, "numRequests": 760690, "compression": { "snappy": { "compressor": { "bytesIn": 0, "bytesOut": 0 }, "decompressor": { "bytesIn": 0, "bytesOut": 0 } } }, "serviceExecutorTaskStats": { "executor": "passthrough", "threadsRunning": 25 } } } -
@djensen47 said in Search crashes MongoDB and NodeBB:
{
"name": "nodebb.searchpost",
"count": 1239083,
"size": 2202184242,
"avgObjSize": 1777,
"storageSize": 835080192,
"totalIndexSize": 2357145600,
"indexSizes": {
"id": 16314368,
"content_text_uid_1_cid_1": 2340831232
}
},Seems like still have 1.2m posts indexed
