MongoDB crashes



  • I have a dedicated MongoDB Server V4.0.10 running on Ubuntu 16.04. (12GB RAM).
    My Nodebb V1.12.2 instance is running a separate server also Ubuntu but v18.10 (96GB RAM).

    I am guessing it happens when my customized RSS plugin is running but it is intermittent.

    Not sure if my DB's integrity is compromised but now I am getting crash dump more frequently (RSS is running every hour). Once it crashes I can not restart until I repair the DB (using --repair command).

    Please advise if anybody has seen similar issues and can share how to reduce the frequency of these crashes or if possible to stabilize it.

    Here are my logs:

    Nodebb:

    2019-07-01T22:01:55.546Z [4566/30410] - verbose: [plugins/fireHook] filter:config.get
    2019-07-01T22:01:55.547Z [4566/30410] - verbose: [plugins/fireHook] filter:middleware.buildHeader
    2019-07-01T22:01:55.547Z [4566/30410] - verbose: [plugins/fireHook] action:plugins.firehook
    2019-07-01T22:01:55.548Z [4566/30410] - verbose: [plugins/fireHook] action:plugins.firehook
    2019-07-01T22:01:55.550Z [4566/30410] - verbose: [plugins/fireHook] action:middleware.pageView
    2019-07-01T22:01:55.550Z [4566/30410] - verbose: [plugins/fireHook] action:plugins.firehook
    2019-07-01T22:01:55.551Z [4566/30410] - verbose: [plugins/fireHook] filter:user.getSettings
    2019-07-01T22:02:04.216Z [4566/30410] - verbose: [plugins/fireHook] filter:config.get
    2019-07-01T22:02:04.216Z [4566/30410] - verbose: [plugins/fireHook] filter:middleware.buildHeader
    2019-07-01T22:02:04.217Z [4566/30410] - verbose: [plugins/fireHook] action:plugins.firehook
    2019-07-01T22:02:04.217Z [4566/30410] - verbose: [plugins/fireHook] action:plugins.firehook
    2019-07-01T22:02:04.218Z [4566/30410] - verbose: [plugins/fireHook] action:middleware.pageView
    2019-07-01T22:02:04.218Z [4566/30410] - verbose: [plugins/fireHook] action:plugins.firehook
    2019-07-01T22:03:44.527Z [4566/30410] - error: [analytics] Encountered error while writing analytics to data storeread ECONNRESET {"name":"MongoNetworkError","errorLabels":["TransientTransactionError"],"stack":"Error: read ECONNRESET\n    at TCP.onStreamRead (internal/stream_base_commons.js:111:27)"}
    2019-07-01T22:03:44.529Z [4566/30410] - verbose: [plugins/fireHook] action:plugins.firehook
    2019-07-01T22:03:44.529Z [4566/30410] - verbose: [plugins/fireHook] filter:user.notificationTypes
    2019-07-01T22:03:44.530Z [4566/30410] - verbose: [plugins/fireHook] action:plugins.firehook
    2019-07-01T22:03:44.530Z [4562/30390] - error: e3-read ECONNRESET {"name":"MongoNetworkError","errorLabels":["TransientTransactionError"],"stack":"Error: read ECONNRESET\n    at TCP.onStreamRead (internal/stream_base_commons.js:111:27)"}
    2019-07-01T22:03:44.536Z [4567/30417] - error: uncaughtException: connection 1 to 192.168.0.100:27017 closed
    MongoError: connection 1 to 192.168.0.100:27017 closed
        at Function.MongoError.create (/home/webapp/nodebb/node_modules/mongodb-core/lib/error.js:29:11)
        at Socket.<anonymous> (/home/webapp/nodebb/node_modules/mongodb-core/lib/connection/connection.js:214:22)
        at Object.onceWrapper (events.js:277:13)
        at Socket.emit (events.js:189:13)
        at TCP._handle.close (net.js:597:12) {"error":{"name":"MongoError","message":"connection 1 to 192.168.0.100:27017 closed"},"stack":"MongoError: connection 1 to 192.168.0.100:27017 closed\n    at Function.MongoError.create (/home/webapp/nodebb/node_modules/mongodb-core/lib/error.js:29:11)\n    at Socket.<anonymous> (/home/webapp/nodebb/node_modules/mongodb-core/lib/connection/connection.js:214:22)\n    at Object.onceWrapper (events.js:277:13)\n    at Socket.emit (events.js:189:13)\n    at TCP._handle.close (net.js:597:12)","exception":true,"date":"Mon Jul 01 2019 18:03:44 GMT-0400 (Eastern Daylight Time)","process":{"pid":30417,"uid":1000,"gid":1000,"cwd":"/home/webapp/nodebb","execPath":"/usr/bin/node","version":"v10.15.3","argv":["/usr/bin/node","/home/webapp/nodebb/app.js"],"memoryUsage":{"rss":196640768,"heapTotal":154943488,"heapUsed":128626608,"external":36445819}},"os":{"loadavg":[0,0,0],"uptime":2258979},"trace":[{"column":11,"file":"/home/webapp/nodebb/node_modules/mongodb-core/lib/error.js","function":"Function.MongoError.create","line":29,"method":"create","native":false},{"column":22,"file":"/home/webapp/nodebb/node_modules/mongodb-core/lib/connection/connection.js","function":null,"line":214,"method":null,"native":false},{"column":13,"file":"events.js","function":"Object.onceWrapper","line":277,"method":"onceWrapper","native":false},{"column":13,"file":"events.js","function":"Socket.emit","line":189,"method":"emit","native":false},{"column":12,"file":"net.js","function":"TCP._handle.close","line":597,"method":"close","native":false}]}
    2019-07-01T22:03:44.536Z [4567/30417] - error: connection 1 to 192.168.0.100:27017 closed {"name":"MongoError"}
    2019-07-01T22:03:44.537Z [4567/30417] - info: [app] Shutdown (SIGTERM/SIGINT) Initialised.
    2019-07-01T22:03:44.537Z [4565/30403] - error: uncaughtException: connection 1 to 192.168.0.100:27017 closed
    MongoNetworkError: connection 1 to 192.168.0.100:27017 closed
        at Socket.<anonymous> (/home/webapp/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:352:9)
        at Object.onceWrapper (events.js:277:13)
        at Socket.emit (events.js:189:13)
        at TCP._handle.close (net.js:597:12) {"error":{"name":"MongoNetworkError","errorLabels":["TransientTransactionError"]},"stack":"MongoNetworkError: connection 1 to 192.168.0.100:27017 closed\n    at Socket.<anonymous> (/home/webapp/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:352:9)\n    at Object.onceWrapper (events.js:277:13)\n    at Socket.emit (events.js:189:13)\n    at TCP._handle.close (net.js:597:12)","exception":true,"date":"Mon Jul 01 2019 18:03:44 GMT-0400 (Eastern Daylight Time)","process":{"pid":30403,"uid":1000,"gid":1000,"cwd":"/home/webapp/nodebb","execPath":"/usr/bin/node","version":"v10.15.3","argv":["/usr/bin/node","/home/webapp/nodebb/app.js"],"memoryUsage":{"rss":194048000,"heapTotal":152846336,"heapUsed":132431528,"external":36512203}},"os":{"loadavg":[0,0,0],"uptime":2258979},"trace":[{"column":9,"file":"/home/webapp/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js","function":null,"line":352,"method":null,"native":false},{"column":13,"file":"events.js","function":"Object.onceWrapper","line":277,"method":"onceWrapper","native":false},{"column":13,"file":"events.js","function":"Socket.emit","line":189,"method":"emit","native":false},{"column":12,"file":"net.js","function":"TCP._handle.close","line":597,"method":"close","native":false}]}
    2019-07-01T22:03:44.537Z [4564/30397] - error: uncaughtException: connection 0 to 192.168.0.100:27017 closed
    MongoError: connection 0 to 192.168.0.100:27017 closed
        at Function.MongoError.create (/home/webapp/nodebb/node_modules/mongodb-core/lib/error.js:29:11)
        at Socket.<anonymous> (/home/webapp/nodebb/node_modules/mongodb-core/lib/connection/connection.js:214:22)
        at Object.onceWrapper (events.js:277:13)
        at Socket.emit (events.js:189:13)
    

    and my mongoDB log is:

    2019-07-01T18:00:51.411-0400 F -        [WTCheckpointThread] Invalid access at address: 0x200000
    2019-07-01T18:00:51.653-0400 F -        [WTCheckpointThread] Got signal: 11 (Segmentation fault).
     0x55bbba2950e1 0x55bbba2942f9 0x55bbba294966 0x7f95ef1d7390 0x55bbb89e40d7 0x55bbb8a4ea1c 0x55bbb8a11e6a 0x55bbb8a14256 0x55bbb8a152a3 0x55bbb89f9f0a 0x55bbb896bfb7 0x55bbb9ce92e1 0x55bbba3a47a0 0x7f95ef1cd6ba 0x7f95eef0341d
    ----- BEGIN BACKTRACE -----
    {"backtrace":[{"b":"55BBB7E73000","o":"24220E1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55BBB7E73000","o":"24212F9"},{"b":"55BBB7E73000","o":"2421966"},{"b":"7F95EF1C6000","o":"11390"},{"b":"55BBB7E73000","o":"B710D7","s":"__wt_reconcile"},{"b":"55BBB7E73000","o":"BDBA1C","s":"__wt_cache_op"},{"b":"55BBB7E73000","o":"B9EE6A"},{"b":"55BBB7E73000","o":"BA1256"},{"b":"55BBB7E73000","o":"BA22A3","s":"__wt_txn_checkpoint"},{"b":"55BBB7E73000","o":"B86F0A"},{"b":"55BBB7E73000","o":"AF8FB7","s":"_ZN5mongo18WiredTigerKVEngine26WiredTigerCheckpointThread3runEv"},{"b":"55BBB7E73000","o":"1E762E1","s":"_ZN5mongo13BackgroundJob7jobBodyEv"},{"b":"55BBB7E73000","o":"25317A0"},{"b":"7F95EF1C6000","o":"76BA"},{"b":"7F95EEDFC000","o":"10741D","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.0.10", "gitVersion" : "c389e7f69f637f7a1ac3cc9fae843b635f20b766", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-151-generic", "version" : "#178-Ubuntu SMP Tue Jun 11 08:30:22 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "55BBB7E73000", "elfType" : 3, "buildId" : "7B09742F7D2C57B8A6D6760D1D96402AA46C2488" }, { "b" : "7FFF819B4000", "elfType" : 3, "buildId" : "046E5EFC10760EF77BD086D8A1F281D60A8D10E8" }, { "b" : "7F95F05D7000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "5C1A06A89F89E1ADAAA507BC5580C0A7931B0AB2" }, { "b" : "7F95F03BC000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "50A923F8DAFECBCD969C8573116A38C18D0E24D5" }, { "b" : "7F95EFF77000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "15FFEB43278726B025F020862BF51302822A40EC" }, { "b" : "7F95EFD0E000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "FF69EA60EBE05F2DD689D2B26FC85A73E5FBC3A0" }, { "b" : "7F95EFB0A000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "37BFC3D8F7E3B022DAC7943B1A5FACD40CEBF0AD" }, { "b" : "7F95EF902000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "69143E8B39040C964D3958490535322675F15DD3" }, { "b" : "7F95EF5F9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "BAD67A84E56E73D031AE507261DA066B35949D34" }, { "b" : "7F95EF3E3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F95EF1C6000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B17C21299099640A6D863E423D99265824E7BB16" }, { "b" : "7F95EEDFC000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "1CA54A6E0D76188105B12E49FE6B8019BF08803A" }, { "b" : "7F95F0846000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "C0ADBAD6F9A33944F2B3567C078EC472A1DAE98E" }, { "b" : "7F95EEBC9000", "path" : "/usr/lib/x86_64-linux-gnu/libidn.so.11", "elfType" : 3, "buildId" : "E09D3783AD1D0BBCD3204FA01E4EF6D756E18F57" }, { "b" : "7F95EE9AD000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "8D1CC1204D6B6D33BD1D2C5A2A0516A2234322CF" }, { "b" : "7F95EE763000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "41971A4A3CCDC54A447F41DF4BD96C948C546E0E" }, { "b" : "7F95EE554000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "8E613D0B8D8E3537785637424782BE8502ABABD2" }, { "b" : "7F95EE303000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "3890D33727391E4A85DC0F819AB0AA29BB5DFC86" }, { "b" : "7F95EE0E9000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "8D9BD4CE26E45EF16075C67D5F5EEAFD8B562832" }, { "b" : "7F95EDDB9000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "17285B5F2BCC671E0A7FA3E29CCD143509B648CD" }, { "b" : "7F95EDB86000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "B11678F560199547DCF726384EA39153EE0DFABF" }, { "b" : "7F95ED950000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "D6B36C5A463EE0FA84FDD6D5FD3F7726EDB90D54" }, { "b" : "7F95ED6D0000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "7B3533D5998D20EE1A1BE3F87789B69041E7F620" }, { "b" : "7F95ED3FE000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "0EEF7058B0737B68BDF89E5DC604D0AC389C8BB1" }, { "b" : "7F95ED1CF000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "FFBA483A43D9EF73925AC116811890C037523DA1" }, { "b" : "7F95ECFCB000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "1E16CB57F699E215A2A8D4EFEF90883BC749B12D" }, { "b" : "7F95ECDC0000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "B789D8D4B4FC333405AB34387D9237F954060EA4" }, { "b" : "7F95ECBA5000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "87783DF8A1058CD150F8886CB36340384093C18F" }, { "b" : "7F95EC964000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "1FE877BE52A424D0636AFD4D35BB330E41D6E0F3" }, { "b" : "7F95EC700000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "A0E2D03FF5CF65937F4425D4EFD4D655243809EB" }, { "b" : "7F95EC4ED000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "E07E186694852D8F69459C6AB28A53F8DA3CE3B6" }, { "b" : "7F95EC2E9000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "3364D4BF2113C4E8D17EF533867ECC99A53413D6" }, { "b" : "7F95EC0E0000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "73A8EADBC85860662B24850E71D4AFBE22C33359" }, { "b" : "7F95EBE56000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "59E742306A4EA2872E061ECCE92F35FADDA75357" }, { "b" : "7F95EBBB4000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "E5C159E415406AE79D21056D752BA949C408B5B1" }, { "b" : "7F95EB981000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "7D15576E1F096614D360784E4A01A1F5FAF908C9" }, { "b" : "7F95EB76B000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "481DB33C28D88E43DA6BED65E1A7599407D4D818" }, { "b" : "7F95EB563000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "9D9C958F1F4894AFEF6AECD90D1C430EA29AC34F" }, { "b" : "7F95EB33A000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "57E25072866B2D30CF02EBE7AE623B84F96FA700" }, { "b" : "7F95EB12B000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "F6F1B4E9F89B716C4A0BA5819BDFFAF4A13EFB91" }, { "b" : "7F95EAEE0000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "C60082E3BB78D0D42868D9B359B89BF66CE5A1A7" }, { "b" : "7F95EAC0B000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "3B0454E57467057071F7AD49651E0FA7B01CF5C7" }, { "b" : "7F95EA9D3000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "FD61CA7A6D603E94E5EFD5C88D8810AE104FCF40" } ] }}
     mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55bbba2950e1]
     mongod(+0x24212F9) [0x55bbba2942f9]
     mongod(+0x2421966) [0x55bbba294966]
     libpthread.so.0(+0x11390) [0x7f95ef1d7390]
     mongod(__wt_reconcile+0x1EA7) [0x55bbb89e40d7]
     mongod(__wt_cache_op+0x5AC) [0x55bbb8a4ea1c]
     mongod(+0xB9EE6A) [0x55bbb8a11e6a]
     mongod(+0xBA1256) [0x55bbb8a14256]
     mongod(__wt_txn_checkpoint+0x1D3) [0x55bbb8a152a3]
     mongod(+0xB86F0A) [0x55bbb89f9f0a]
     mongod(_ZN5mongo18WiredTigerKVEngine26WiredTigerCheckpointThread3runEv+0x697) [0x55bbb896bfb7]
     mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x131) [0x55bbb9ce92e1]
     mongod(+0x25317A0) [0x55bbba3a47a0]
     libpthread.so.0(+0x76BA) [0x7f95ef1cd6ba]
     libc.so.6(clone+0x6D) [0x7f95eef0341d]
    -----  END BACKTRACE  -----
    Segmentation fault (core dumped)
    

    Please note that my test/development HW is same configuration as MongoDB but running WIN-10 and I have both Nodebb and MongoDB co-located. I have not had experience any any issues on my development only in production.

    Any suggestion is highly appreciated.


Log in to reply
 

Suggested Topics

| |