Topics create 20X records in DB



  • Right now my team is working with NodeBB for a content-oriented site. We have very few users in the system during development but we're creating a lot of topics via automated posting of content from the web.

    With only 100K topics in the system (nearly all with a single post) we have around 2M records in the database. Distinct _key in the database is around 1/5 of the total records (400K).

    Part of this appears to be from tags - where an average post will have 3-4 tags associated with it and it looks like there is a record created for each tag + each post.

    Is there anything we can do to slow the hyper-growth in database documents? I'm concerned the number of records being created will grow exponentially further when we have more users in the system - with records for tracking per-user notifications, and read status of each topic/post, etc.

    It would be great to get some advice on how to mitigate this issue, including:

    • Which admin settings control record creation
    • Which user/profile settings control record creation
    • Which database records can be safely pruned after disabling specific features
    • Etc.

    Thanks!

    PS: We're currently using 0.7.x with custom plugins/theme and MongoDB as the database layer.



  • This is because how we store data. For example each sorted set element is stored as document.

    For example

    db.objects.find({_key:"topics:tid"});
    { "_id" : ObjectId("5547aee465190fe212295e96"), "_key" : "topics:tid", "value" : "4392", "score" : 1430511213134 }
    { "_id" : ObjectId("5547aee465190fe212295e94"), "_key" : "topics:tid", "value" : "4391", "score" : 1430500193058 }
    { "_id" : ObjectId("5547aee465190fe212295e92"), "_key" : "topics:tid", "value" : "4390", "score" : 1430494440461 }
    { "_id" : ObjectId("5547aee465190fe212295e90"), "_key" : "topics:tid", "value" : "4389", "score" : 1430484949083 }
    { "_id" : ObjectId("5547aee465190fe212295e8f"), "_key" : "topics:tid", "value" : "4388", "score" : 1430479608551 }
    { "_id" : ObjectId("5547aee465190fe212295e8e"), "_key" : "topics:tid", "value" : "4387", "score" : 1430435412827 }
    

    When you post a topic it's topic id and data is stored in many places. For example under topics:recent, topics:tid, user:1:tids and so on.


Log in to reply
 

Suggested Topics

  • 2
  • 8
  • 4
  • 4
  • 1
| |