I have a dedicated MongoDB Server V4.0.10 running on Ubuntu 16.04. (12GB RAM).
My Nodebb V1.12.2 instance is running a separate server also Ubuntu but v18.10 (96GB RAM).
I am guessing it happens when my customized RSS plugin is running but it is intermittent.
Not sure if my DB's integrity is compromised but now I am getting crash dump more frequently (RSS is running every hour). Once it crashes I can not restart until I repair the DB (using --repair command).
Please advise if anybody has seen similar issues and can share how to reduce the frequency of these crashes or if possible to stabilize it.
Here are my logs:
Nodebb:
2019-07-01T22:01:55.546Z [4566/30410] - verbose: [plugins/fireHook] filter:config.get
2019-07-01T22:01:55.547Z [4566/30410] - verbose: [plugins/fireHook] filter:middleware.buildHeader
2019-07-01T22:01:55.547Z [4566/30410] - verbose: [plugins/fireHook] action:plugins.firehook
2019-07-01T22:01:55.548Z [4566/30410] - verbose: [plugins/fireHook] action:plugins.firehook
2019-07-01T22:01:55.550Z [4566/30410] - verbose: [plugins/fireHook] action:middleware.pageView
2019-07-01T22:01:55.550Z [4566/30410] - verbose: [plugins/fireHook] action:plugins.firehook
2019-07-01T22:01:55.551Z [4566/30410] - verbose: [plugins/fireHook] filter:user.getSettings
2019-07-01T22:02:04.216Z [4566/30410] - verbose: [plugins/fireHook] filter:config.get
2019-07-01T22:02:04.216Z [4566/30410] - verbose: [plugins/fireHook] filter:middleware.buildHeader
2019-07-01T22:02:04.217Z [4566/30410] - verbose: [plugins/fireHook] action:plugins.firehook
2019-07-01T22:02:04.217Z [4566/30410] - verbose: [plugins/fireHook] action:plugins.firehook
2019-07-01T22:02:04.218Z [4566/30410] - verbose: [plugins/fireHook] action:middleware.pageView
2019-07-01T22:02:04.218Z [4566/30410] - verbose: [plugins/fireHook] action:plugins.firehook
2019-07-01T22:03:44.527Z [4566/30410] - error: [analytics] Encountered error while writing analytics to data storeread ECONNRESET {"name":"MongoNetworkError","errorLabels":["TransientTransactionError"],"stack":"Error: read ECONNRESET\n at TCP.onStreamRead (internal/stream_base_commons.js:111:27)"}
2019-07-01T22:03:44.529Z [4566/30410] - verbose: [plugins/fireHook] action:plugins.firehook
2019-07-01T22:03:44.529Z [4566/30410] - verbose: [plugins/fireHook] filter:user.notificationTypes
2019-07-01T22:03:44.530Z [4566/30410] - verbose: [plugins/fireHook] action:plugins.firehook
2019-07-01T22:03:44.530Z [4562/30390] - error: e3-read ECONNRESET {"name":"MongoNetworkError","errorLabels":["TransientTransactionError"],"stack":"Error: read ECONNRESET\n at TCP.onStreamRead (internal/stream_base_commons.js:111:27)"}
2019-07-01T22:03:44.536Z [4567/30417] - error: uncaughtException: connection 1 to 192.168.0.100:27017 closed
MongoError: connection 1 to 192.168.0.100:27017 closed
at Function.MongoError.create (/home/webapp/nodebb/node_modules/mongodb-core/lib/error.js:29:11)
at Socket.<anonymous> (/home/webapp/nodebb/node_modules/mongodb-core/lib/connection/connection.js:214:22)
at Object.onceWrapper (events.js:277:13)
at Socket.emit (events.js:189:13)
at TCP._handle.close (net.js:597:12) {"error":{"name":"MongoError","message":"connection 1 to 192.168.0.100:27017 closed"},"stack":"MongoError: connection 1 to 192.168.0.100:27017 closed\n at Function.MongoError.create (/home/webapp/nodebb/node_modules/mongodb-core/lib/error.js:29:11)\n at Socket.<anonymous> (/home/webapp/nodebb/node_modules/mongodb-core/lib/connection/connection.js:214:22)\n at Object.onceWrapper (events.js:277:13)\n at Socket.emit (events.js:189:13)\n at TCP._handle.close (net.js:597:12)","exception":true,"date":"Mon Jul 01 2019 18:03:44 GMT-0400 (Eastern Daylight Time)","process":{"pid":30417,"uid":1000,"gid":1000,"cwd":"/home/webapp/nodebb","execPath":"/usr/bin/node","version":"v10.15.3","argv":["/usr/bin/node","/home/webapp/nodebb/app.js"],"memoryUsage":{"rss":196640768,"heapTotal":154943488,"heapUsed":128626608,"external":36445819}},"os":{"loadavg":[0,0,0],"uptime":2258979},"trace":[{"column":11,"file":"/home/webapp/nodebb/node_modules/mongodb-core/lib/error.js","function":"Function.MongoError.create","line":29,"method":"create","native":false},{"column":22,"file":"/home/webapp/nodebb/node_modules/mongodb-core/lib/connection/connection.js","function":null,"line":214,"method":null,"native":false},{"column":13,"file":"events.js","function":"Object.onceWrapper","line":277,"method":"onceWrapper","native":false},{"column":13,"file":"events.js","function":"Socket.emit","line":189,"method":"emit","native":false},{"column":12,"file":"net.js","function":"TCP._handle.close","line":597,"method":"close","native":false}]}
2019-07-01T22:03:44.536Z [4567/30417] - error: connection 1 to 192.168.0.100:27017 closed {"name":"MongoError"}
2019-07-01T22:03:44.537Z [4567/30417] - info: [app] Shutdown (SIGTERM/SIGINT) Initialised.
2019-07-01T22:03:44.537Z [4565/30403] - error: uncaughtException: connection 1 to 192.168.0.100:27017 closed
MongoNetworkError: connection 1 to 192.168.0.100:27017 closed
at Socket.<anonymous> (/home/webapp/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:352:9)
at Object.onceWrapper (events.js:277:13)
at Socket.emit (events.js:189:13)
at TCP._handle.close (net.js:597:12) {"error":{"name":"MongoNetworkError","errorLabels":["TransientTransactionError"]},"stack":"MongoNetworkError: connection 1 to 192.168.0.100:27017 closed\n at Socket.<anonymous> (/home/webapp/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:352:9)\n at Object.onceWrapper (events.js:277:13)\n at Socket.emit (events.js:189:13)\n at TCP._handle.close (net.js:597:12)","exception":true,"date":"Mon Jul 01 2019 18:03:44 GMT-0400 (Eastern Daylight Time)","process":{"pid":30403,"uid":1000,"gid":1000,"cwd":"/home/webapp/nodebb","execPath":"/usr/bin/node","version":"v10.15.3","argv":["/usr/bin/node","/home/webapp/nodebb/app.js"],"memoryUsage":{"rss":194048000,"heapTotal":152846336,"heapUsed":132431528,"external":36512203}},"os":{"loadavg":[0,0,0],"uptime":2258979},"trace":[{"column":9,"file":"/home/webapp/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js","function":null,"line":352,"method":null,"native":false},{"column":13,"file":"events.js","function":"Object.onceWrapper","line":277,"method":"onceWrapper","native":false},{"column":13,"file":"events.js","function":"Socket.emit","line":189,"method":"emit","native":false},{"column":12,"file":"net.js","function":"TCP._handle.close","line":597,"method":"close","native":false}]}
2019-07-01T22:03:44.537Z [4564/30397] - error: uncaughtException: connection 0 to 192.168.0.100:27017 closed
MongoError: connection 0 to 192.168.0.100:27017 closed
at Function.MongoError.create (/home/webapp/nodebb/node_modules/mongodb-core/lib/error.js:29:11)
at Socket.<anonymous> (/home/webapp/nodebb/node_modules/mongodb-core/lib/connection/connection.js:214:22)
at Object.onceWrapper (events.js:277:13)
at Socket.emit (events.js:189:13)
and my mongoDB log is:
2019-07-01T18:00:51.411-0400 F - [WTCheckpointThread] Invalid access at address: 0x200000
2019-07-01T18:00:51.653-0400 F - [WTCheckpointThread] Got signal: 11 (Segmentation fault).
0x55bbba2950e1 0x55bbba2942f9 0x55bbba294966 0x7f95ef1d7390 0x55bbb89e40d7 0x55bbb8a4ea1c 0x55bbb8a11e6a 0x55bbb8a14256 0x55bbb8a152a3 0x55bbb89f9f0a 0x55bbb896bfb7 0x55bbb9ce92e1 0x55bbba3a47a0 0x7f95ef1cd6ba 0x7f95eef0341d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55BBB7E73000","o":"24220E1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55BBB7E73000","o":"24212F9"},{"b":"55BBB7E73000","o":"2421966"},{"b":"7F95EF1C6000","o":"11390"},{"b":"55BBB7E73000","o":"B710D7","s":"__wt_reconcile"},{"b":"55BBB7E73000","o":"BDBA1C","s":"__wt_cache_op"},{"b":"55BBB7E73000","o":"B9EE6A"},{"b":"55BBB7E73000","o":"BA1256"},{"b":"55BBB7E73000","o":"BA22A3","s":"__wt_txn_checkpoint"},{"b":"55BBB7E73000","o":"B86F0A"},{"b":"55BBB7E73000","o":"AF8FB7","s":"_ZN5mongo18WiredTigerKVEngine26WiredTigerCheckpointThread3runEv"},{"b":"55BBB7E73000","o":"1E762E1","s":"_ZN5mongo13BackgroundJob7jobBodyEv"},{"b":"55BBB7E73000","o":"25317A0"},{"b":"7F95EF1C6000","o":"76BA"},{"b":"7F95EEDFC000","o":"10741D","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.0.10", "gitVersion" : "c389e7f69f637f7a1ac3cc9fae843b635f20b766", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-151-generic", "version" : "#178-Ubuntu SMP Tue Jun 11 08:30:22 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "55BBB7E73000", "elfType" : 3, "buildId" : "7B09742F7D2C57B8A6D6760D1D96402AA46C2488" }, { "b" : "7FFF819B4000", "elfType" : 3, "buildId" : "046E5EFC10760EF77BD086D8A1F281D60A8D10E8" }, { "b" : "7F95F05D7000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "5C1A06A89F89E1ADAAA507BC5580C0A7931B0AB2" }, { "b" : "7F95F03BC000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "50A923F8DAFECBCD969C8573116A38C18D0E24D5" }, { "b" : "7F95EFF77000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "15FFEB43278726B025F020862BF51302822A40EC" }, { "b" : "7F95EFD0E000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "FF69EA60EBE05F2DD689D2B26FC85A73E5FBC3A0" }, { "b" : "7F95EFB0A000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "37BFC3D8F7E3B022DAC7943B1A5FACD40CEBF0AD" }, { "b" : "7F95EF902000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "69143E8B39040C964D3958490535322675F15DD3" }, { "b" : "7F95EF5F9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "BAD67A84E56E73D031AE507261DA066B35949D34" }, { "b" : "7F95EF3E3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F95EF1C6000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B17C21299099640A6D863E423D99265824E7BB16" }, { "b" : "7F95EEDFC000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "1CA54A6E0D76188105B12E49FE6B8019BF08803A" }, { "b" : "7F95F0846000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "C0ADBAD6F9A33944F2B3567C078EC472A1DAE98E" }, { "b" : "7F95EEBC9000", "path" : "/usr/lib/x86_64-linux-gnu/libidn.so.11", "elfType" : 3, "buildId" : "E09D3783AD1D0BBCD3204FA01E4EF6D756E18F57" }, { "b" : "7F95EE9AD000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "8D1CC1204D6B6D33BD1D2C5A2A0516A2234322CF" }, { "b" : "7F95EE763000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "41971A4A3CCDC54A447F41DF4BD96C948C546E0E" }, { "b" : "7F95EE554000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "8E613D0B8D8E3537785637424782BE8502ABABD2" }, { "b" : "7F95EE303000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "3890D33727391E4A85DC0F819AB0AA29BB5DFC86" }, { "b" : "7F95EE0E9000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "8D9BD4CE26E45EF16075C67D5F5EEAFD8B562832" }, { "b" : "7F95EDDB9000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "17285B5F2BCC671E0A7FA3E29CCD143509B648CD" }, { "b" : "7F95EDB86000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "B11678F560199547DCF726384EA39153EE0DFABF" }, { "b" : "7F95ED950000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "D6B36C5A463EE0FA84FDD6D5FD3F7726EDB90D54" }, { "b" : "7F95ED6D0000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "7B3533D5998D20EE1A1BE3F87789B69041E7F620" }, { "b" : "7F95ED3FE000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "0EEF7058B0737B68BDF89E5DC604D0AC389C8BB1" }, { "b" : "7F95ED1CF000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "FFBA483A43D9EF73925AC116811890C037523DA1" }, { "b" : "7F95ECFCB000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "1E16CB57F699E215A2A8D4EFEF90883BC749B12D" }, { "b" : "7F95ECDC0000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "B789D8D4B4FC333405AB34387D9237F954060EA4" }, { "b" : "7F95ECBA5000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "87783DF8A1058CD150F8886CB36340384093C18F" }, { "b" : "7F95EC964000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "1FE877BE52A424D0636AFD4D35BB330E41D6E0F3" }, { "b" : "7F95EC700000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "A0E2D03FF5CF65937F4425D4EFD4D655243809EB" }, { "b" : "7F95EC4ED000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "E07E186694852D8F69459C6AB28A53F8DA3CE3B6" }, { "b" : "7F95EC2E9000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "3364D4BF2113C4E8D17EF533867ECC99A53413D6" }, { "b" : "7F95EC0E0000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "73A8EADBC85860662B24850E71D4AFBE22C33359" }, { "b" : "7F95EBE56000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "59E742306A4EA2872E061ECCE92F35FADDA75357" }, { "b" : "7F95EBBB4000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "E5C159E415406AE79D21056D752BA949C408B5B1" }, { "b" : "7F95EB981000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "7D15576E1F096614D360784E4A01A1F5FAF908C9" }, { "b" : "7F95EB76B000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "481DB33C28D88E43DA6BED65E1A7599407D4D818" }, { "b" : "7F95EB563000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "9D9C958F1F4894AFEF6AECD90D1C430EA29AC34F" }, { "b" : "7F95EB33A000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "57E25072866B2D30CF02EBE7AE623B84F96FA700" }, { "b" : "7F95EB12B000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "F6F1B4E9F89B716C4A0BA5819BDFFAF4A13EFB91" }, { "b" : "7F95EAEE0000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "C60082E3BB78D0D42868D9B359B89BF66CE5A1A7" }, { "b" : "7F95EAC0B000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "3B0454E57467057071F7AD49651E0FA7B01CF5C7" }, { "b" : "7F95EA9D3000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "FD61CA7A6D603E94E5EFD5C88D8810AE104FCF40" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55bbba2950e1]
mongod(+0x24212F9) [0x55bbba2942f9]
mongod(+0x2421966) [0x55bbba294966]
libpthread.so.0(+0x11390) [0x7f95ef1d7390]
mongod(__wt_reconcile+0x1EA7) [0x55bbb89e40d7]
mongod(__wt_cache_op+0x5AC) [0x55bbb8a4ea1c]
mongod(+0xB9EE6A) [0x55bbb8a11e6a]
mongod(+0xBA1256) [0x55bbb8a14256]
mongod(__wt_txn_checkpoint+0x1D3) [0x55bbb8a152a3]
mongod(+0xB86F0A) [0x55bbb89f9f0a]
mongod(_ZN5mongo18WiredTigerKVEngine26WiredTigerCheckpointThread3runEv+0x697) [0x55bbb896bfb7]
mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x131) [0x55bbb9ce92e1]
mongod(+0x25317A0) [0x55bbba3a47a0]
libpthread.so.0(+0x76BA) [0x7f95ef1cd6ba]
libc.so.6(clone+0x6D) [0x7f95eef0341d]
----- END BACKTRACE -----
Segmentation fault (core dumped)
Please note that my test/development HW is same configuration as MongoDB but running WIN-10 and I have both Nodebb and MongoDB co-located. I have not had experience any any issues on my development only in production.
Any suggestion is highly appreciated.
