• Home
  • Categories
  • Recent
  • Popular
  • Top
  • Tags
  • Users
  • Groups
  • Documentation
    • Home
    • Read API
    • Write API
    • Plugin Development
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
v3.5.2 Latest
Buy Hosting

Infinite loop

Scheduled Pinned Locked Moved Unsolved Technical Support
10 Posts 2 Posters 3.6k Views
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • Ben LubarB Offline
    Ben LubarB Offline
    Ben Lubar Gamers
    wrote on last edited by
    #1

    On what.thedailywtf.com, we've been having trouble with the NodeJS instances stopping handling requests and running at near 100% CPU usage until they are restarted. We run MongoDB for the main database, Redis for clustering support, and 2 NodeBB clustered instances (down from 4 due to this problem).

    At this point, I have no idea what could be causing the problem, other than that it started about a week ago. Is there some way to get a stack trace from a running NodeJS instance?

    There's no log output during the time the problem occurs. servercooties.com tracks our instance's uptime, so @accalia can probably pinpoint the times when the problem occurred.

    /cc @boomzilla @PJH

    1 Reply Last reply
    1
  • barisB Offline
    barisB Offline
    <baris> NodeBB
    wrote on last edited by
    #2

    Are you using nginx to direct traffic to nodebb? Are there any errors in the nginx logs?

    Ben LubarB 1 Reply Last reply
    0
  • Ben LubarB Offline
    Ben LubarB Offline
    Ben Lubar Gamers
    replied to <baris> on last edited by
    #3

    @baris said in Infinite loop:

    Are you using nginx to direct traffic to nodebb?

    yes

    @baris said in Infinite loop:

    Are there any errors in the nginx logs?

    $ wc -l /var/log/nginx/error.log
    180933 /var/log/nginx/error.log
    

    Anything specific I should look for?

    1 Reply Last reply
    0
  • barisB Offline
    barisB Offline
    <baris> NodeBB
    wrote on last edited by baris
    #4

    Look for any socket.io errors if you have alot of those you can change the upstream block for nodebb to the below.

    upstream nodebb{
        ip_hash;
        server 127.0.0.1:4567 max_fails=0 fail_timeout=10s; 
        server 127.0.0.1:4568 max_fails=0 fail_timeout=10s; 
        keepalive 512;
    }
    

    This helps under highload if socket.io is throwing alot of errors .

    1 Reply Last reply
    0
  • Ben LubarB Offline
    Ben LubarB Offline
    Ben Lubar Gamers
    wrote on last edited by Ben Lubar
    #5

    ok, I see that there is a way to get a stack trace from NodeJS, but it requires starting it with --debug-brk so a debugger can be attached (and then unpaused and allowed to run as normal until it hangs).

    Should I edit loader.js for that, or is there some other way to start one of the instances of a cluster with an extra parameter?

    barisB 1 Reply Last reply
    0
  • barisB Offline
    barisB Offline
    <baris> NodeBB
    replied to Ben Lubar on last edited by
    #6

    @Ben-Lubar Your best bet would be to edit loader.js, I remember trying to pause a running instance to get a stack trace long ago but didn't succeed. Let me know if you have better luck.

    Ben LubarB 1 Reply Last reply
    0
  • Ben LubarB Offline
    Ben LubarB Offline
    Ben Lubar Gamers
    replied to <baris> on last edited by
    #7

    @baris I was able to pause it and get a backtrace up until it actually encountered the hang, and then the debugger never finished whatever command I threw at it.

    I'm stumped.

    1 Reply Last reply
    0
  • barisB Offline
    barisB Offline
    <baris> NodeBB
    wrote on last edited by
    #8

    I would look at anything that is different from a default install, ie custom plugins, core modifications, changes to default settings like increasing unread cutoff time etc.

    We are not experiencing the hangs on this site or busy sites that we host so it could be a configuration change that's causing it.

    1 Reply Last reply
    0
  • Ben LubarB Offline
    Ben LubarB Offline
    Ben Lubar Gamers
    wrote on last edited by
    #9

    Okay, strace showed both instances hanging after this:

    connect(85, {sa_family=AF_INET, sin_port=htons("[iframely port]"), sin_addr=inet_addr("[iframely IP]")}, 16) = -1 EINPROGRESS (Operation now in progress)
    

    I've blacklisted the forum's hostname from iframely. We'll see if that fixes it.

    1 Reply Last reply
    2
  • Ben LubarB Offline
    Ben LubarB Offline
    Ben Lubar Gamers
    wrote on last edited by
    #10

    Ok, time for an update. The iframely problem was definitely one of the causes, but we're still getting occasional lock-ups. Nowhere near as bad as before, but the site still goes down a few times a day. We have a topic where we track manual restarts, but the data doesn't really show any pattern other than that the infinite loop is happening during times when a lot of pages are loaded.

    1 Reply Last reply
    0

Copyright © 2023 NodeBB | Contributors
  • Login

  • Don't have an account? Register

  • Login or register to search.
Powered by NodeBB Contributors
  • First post
    Last post
0
  • Home
  • Categories
  • Recent
  • Popular
  • Top
  • Tags
  • Users
  • Groups
  • Documentation
    • Home
    • Read API
    • Write API
    • Plugin Development