Debug Nodebb load

Technical Support
  • Hello guys,
    I'm looking for the ideas how to debug nodebb load.

    Right now our nodebb application is using 200 registered users (400 connections with guests) online and we have about 1 million topics.
    Unfortunately, our community receiving very often 503 (heavy load) error page, and application is very slow.

    Our stack looks like:

    • haproxy loadbalancer
    • varnish (cache statics)
    • nodebb loadbalancer
    • 10 - 20 docker instances of nodebb running on two ports each

    Everything of it is working on 3 hosts, from 13 to 27GB RAM and about 10% cpu load.

    Where should we start to debug why nodebb is slow?

    @julian @psychobunny

  • @damian-gądziak what does /admin/developmnent/info show? Are users distributed to backends equally? Are you serving static assets with your webserver?

  • What database are you using? Have you tried running fewer instances?

  • @PitaJ we are now testing 10 instances instead of 20
    @baris great tip! I didn't saw before this debug page, we don't have it visible in admin menu (nodebb 1.5.1). Now I see that only half of our instances getting the traffic - so there is a problem with loadbalancer.
    We are serving static files via varnish cache.

    Should we expect a huge performance difference between version 1.5 and 1.8 of nodebb?

  • @damian-gądziak I believe performance should be better check your package.json for templates.js, we switched from that to benchpress which is alot faster. There were alot of cache improvements on later versions as well.

  • @baris Performace is much much better for guests, but other hands worse for signed in users.
    We have just upgraded nodebb from 1.5.1 to 1.8.2.

    A guest visitor is able to get HTTP response from the homepage in 100-200ms, where signed in user need to wait for about 4-5s.
    We have already 7M topics (not 1M) and probably some redis queries are not optimized

  • @damian-gądziak Is your forum public? Is the hompage loading a list of categories? What is the size of the payload when you load /api for guests vs registered users?

  • @baris we are not loading categories at home page, there is even no request to nodebb /api endpoint (as a guest).
    Yes, it's public ( you can compare page speed as a guest and signed in user.
    There is 7M topic because we have about 7M subpages connected via nodebb-plugin-comments

  • There is also one more application entry via (this domain has turned ON varnish for static files), is serving statics via nodebb as now, but it seems to not be a clue of the problem

  • Not sure how your homepage is rendered, since both guests and registered users get the same response in /api, I would check the code that you have for registered-users and figure out why it is taking longer than guests. Number of topics in the database should not effect the load speed unless there is some code that is doing a lot of processing on those topics to load the homepage.

  • @baris it seems that signed in user need some data from redis, but after nodebb upgrade redis stats looks not so good,
    total operations increased from 9M to 25M per minute
    CPU load from 40% to 100%
    so yeach, that's the problem

  • If you have custom code to load the homepage take a look there and see if anything is being loaded in a loop. You can post here if you can maybe we can spot some inefficiency.

  • @baris at redis monitor I see a lot queries like:
    "zscore" "uid:41843:followed_tids" "990614"

    What they mean ? User #41843 watching topic #990614 ?

  • So, interesting thing.
    I have disabled execution of this query

    "zscore" "uid:41843:followed_tids" "990614"

    by hardcoding response of "Topics.filterWatchedTids" function (always return []).

    Redis and CPU load decreased of about 50%

    Next query on my list is:

      1. (integer) 369411
    1. (integer) 1524255134
      3) (integer) 291408
      4) 1) "zrevrangebyscore"
      2) "topics:recent"
      3) "+inf"
      4) "1524082333364"
      5) "WITHSCORES"
      6) "LIMIT"
      7) "0"
      😎 "-1"

    which took about 0.3s to run and basically should be cached

  • @damian-gądziak Maybe you can try to debug by adding this in that function console.log(tids.length, uid, new Error('test').stack);

    	Topics.filterWatchedTids = function (tids, uid, callback) {
                    console.log(tids.length, uid, new Error('test').stack);
    			function (next) {
    				db.sortedSetScores('uid:' + uid + ':followed_tids', tids, next);
    			function (scores, next) {
    				tids = tids.filter(function (tid, index) {
    					return tid && !!scores[index];
    				next(null, tids);
    		], callback);

    I am guessing the function is called by a huge amount of tids, so the next step would be to find out why and from where.

Suggested Topics