Posting to a Specific Thread Completely Locks Up NodeBB

  • NodeBB

    Weird, does it happen in this thread as well? http://mangolassi.it/topic/1022/what-are-you-doing-right-now

    You can also try these http://docs.mongodb.org/manual/administration/monitoring/ to see what the database is doing during the spike.

    I see you are on 0.5.0-2 now is it still happening on that version?

  • Community Rep

    Yes on 0.5.0-2

    No, that other thread is fine. I've been watching it carefully assuming that it would freak out too but it doesn't. There is something unique about that one thread.

    I will wait until we have a lull (site is active around the clock) and see what the DB is doing.

    I suspect one of the posts contains something causing a problem. But who knows what 😞

  • Community Rep

    We have noticed that even reading the thread causing the CPU to spike for a few minutes.


  • @scottalanmiller Strange, I've got a topic with just under 600 posts on my forum, and it loads and I can post to it without issue (I do use Pagination though). Does it happen when you view a certain portion of posts? Maybe one of the posts is causing the issue?

  • NodeBB

    I am seeing this in one of the posts and a matching error in the browser console.

    <p><a href="uploading 10%" target="_blank"><img src="uploading 10%" alt="Awesome recovery.gif" class="img-responsive"></a> </p>

    GET http://mangolassi.it/topic/389/random-thread-anything-goes/uploading%2010% 504 (Gateway Time-out) nodebb.min.js?eb0c6f65:3

    Maybe thats related.

    It causes this on the server side

    TypeError: Failed to decode param 'uploading%2010%'
        at decode_param (/home/baris/node-forum/node_modules/express/lib/router/layer.js:147:15)
        at Layer.match (/home/baris/node-forum/node_modules/express/lib/router/layer.js:119:11)
        at Function.match_layer (/home/baris/node-forum/node_modules/express/lib/router/index.js:281:16)
        at next (/home/baris/node-forum/node_modules/express/lib/router/index.js:182:10)
        at /home/baris/node-forum/node_modules/express/lib/router/index.js:184:16
        at Function.match_layer (/home/baris/node-forum/node_modules/express/lib/router/index.js:288:3)
        at next (/home/baris/node-forum/node_modules/express/lib/router/index.js:182:10)
        at /home/baris/node-forum/node_modules/express/lib/router/index.js:184:16
        at Function.match_layer (/home/baris/node-forum/node_modules/express/lib/router/index.js:288:3)
        at next (/home/baris/node-forum/node_modules/express/lib/router/index.js:182:10)
    
  • Community Rep

    @a_5mith said:

    @scottalanmiller Strange, I've got a topic with just under 600 posts on my forum, and it loads and I can post to it without issue (I do use Pagination though). Does it happen when you view a certain portion of posts? Maybe one of the posts is causing the issue?

    Yes, I think that it is a single post, we've got others nearly as long. It didn't deteriorate, it failed suddenly.

  • Community Rep

    @baris said:

    I am seeing this in one of the posts and a matching error in the browser console.

    So maybe we have something being treated as non-text causing an issue? That would make sense.

  • Community Rep

    @baris said:

    Awesome recovery.gif

    So how do I edit that entry so that it doesn't crash? Deleting it won't make it completely go aware. But I can't delete or edit anyway because just by going to the page makes the server hang. Hmm...

  • NodeBB

    Hmm once you are on that page you can inspect that element to get its post id. Once you have that you can update it in the database itself it will have a _key = post:<post id>.

    db.objects.find({_key:"post:<post_id>");

    Then something like

    db.objects.update({_key:"post:<post_id>"}, {$set:{"content": "testing edit"}});

    Let me know if that works.

  • NodeBB

    Any updates on this?

    I am seeing a similar issue on a pretty busy mongo instance, when it happens the nodebb nodejs process hits 100% cpu but the mongo process is idle. Usually it takes a restart to fix it.

  • Community Rep

    This issue has returned for us big time. We don't know what thread is doing it now but it is happening about once an hour. It is getting really bad all of a sudden - of course coinciding with a major increase of traffic over the last two days. We are getting a little desperate now.

  • Community Rep

    @baris said:

    Any updates on this?

    I am seeing a similar issue on a pretty busy mongo instance, when it happens the nodebb nodejs process hits 100% cpu but the mongo process is idle. Usually it takes a restart to fix it.

    that's exactly what we see. We now just hop in and do an immediate "kill -9" on the offending process and restart right away. It is horrible.

  • Community Rep

    Oh sure, it happened while I was posting this! That's twice in ten minutes now!

  • Admin

    I'll wait for @baris reply but since he replied to you a month ago he's done a ton of improvements to the mongodb driver (our biggest client is actually using mongo right now) so hang tight, hopefully we can help. Are you on the latest code at the moment (0.5.1)?

  • Community Rep

    No, we are on 0.5.0-4. When should we update?

  • NodeBB

    You can update to 0.5.1, when it happens can you send a screen grab of the top and mongostat that might give us more insight.

  • Community Rep

    @baris said:

    You can update to 0.5.1, when it happens can you send a screen grab of the top and mongostat that might give us more insight.

    Will do.

  • NodeBB

    Also after updating you can drop all indexes on the objects collection see here for more info.

  • Community Rep

    Update completed. Watching for the issues now.

  • Community Rep

    Insanely busy day (topping 1200 hits per hour) and no issues. If this had been yesterday, we would have crashed nearly twenty times! Things are looking much, much better.

Suggested Topics

| |