Database Mystery (Mongo)

TaLoche

Hello everyone,

I've had my nodebb forum since 2014 and I've been updating it since then. I haven't had lately the time to follow up on the latest updates, so my forum was still stuck on v1.10.1.
Since v.1.13.0 was just released I thought it would be nice to do a new setup for my nodebb forum since it was starting to get a bit bloated (around 10Go).

So I did a mongo dump, started a fresh droplet on digitalocean and did a new setup with Ubuntu 18.04 and mongoDB 4.0.13 then did a mongorestore.

Everything worked fine, I had a bit of trouble because of plugins, so in the end, rather then setting up a brand new v1.13.0 nodebb forum, I started with a brand new v1.10.1 nodebb setup and did the mongorestore then. And went on updating it to v1.12.2.

I'm stilling having a few issues with plugins but my main concern is when I hit the "New Topic" button, either on the homepage or inside a section.
I'm able to start writing a first post and hit the publish button as usual. But when I go back to this new topic, I end up with my first post being linked with old messages beneath it. Each new topic I start now has 19 old messages attached to it. If I delete them and refresh, 19 new old messages will be there, staring at me like an old woman.

Do you have an idea what went wrong or what to do? Since I have this issue, I have disabled all my plugins but this strange behaviour is still there.

Thanks for any help

And kudos to the nodeBB Team for the quality updates throughout the years (yes, I'm sucking up to them)

help me obiwan

<baris>

Can you share the /api/topic/ output of one of the topic that has this problem? Do all topics have this issue?

TaLoche

Hi @baris thanks for your help.

It's very strange, if I create a new category, it will either take the spot of an existing category with all the topics existing beforehand now linked to this new category.

Or I can create a new category, create a topic inside and end up having the 19 old messages linked to my first post bug.

This weird behaviour happens when I create a new topic in an existing category or a new one. If I reply to an existing thread, everything works fine.

I have the impression that categories ID are all messed up. Sometimes when creating a new category it will have a cover image from an older version of my forum.

Here's all the information I can gather:

Database Setup

https://i.imgur.com/ncwoStz.png

The bug in motion

https://gifyu.com/image/vday

Regarding your /api/topic/ request

here's the JSON file for the topic created on the gif https://api.myjson.com/bins/1gydvy

Thanks again!

<baris>

@TaLoche said in Database Mystery (Mongo):

It's very strange, if I create a new category, it will either take the spot of an existing category with all the topics existing beforehand now linked to this new category.

This is just a guess, but this sounds like the global object that keeps track of the next category Id got messed up somehow. So when you create a new category instead of getting the next id it started from 1 again and replaced existing categories. You can check the contents of it with db.objects.findOne({_key: "global"}); and then check the existing categories. The nextCid field in the global object should be greater than all your existing category ids.

TaLoche

@baris Thanks a lot for your idea. I'll try to find some time this afternoon to see if I can dig out something useful along the lines you've suggested. I'll keep the topic updated with my progress.

TaLoche

@baris My highest ID as category was 17, and when I created a new one, the ID was 11. So I created fake categories to get up to number 18 and the category was finally empty when I created it. But as soon as I posted a new topic inside, old posts where linked to this first new post.
When I did the mongorestore should I have dropped the existing database (it was empty and only had the admin user account)?

Here is what I get

> db.objects.findOne({_key: "global"});
{
        "_id" : ObjectId("5dd298e8eb7500a3dd086e72"),
        "_key" : "global",
        "nextEid" : 202,
        "nextCid" : 18,
        "nextUid" : 1,
        "userCount" : 1,
        "nextTid" : 18,
        "topicCount" : 15,
        "nextPid" : 24,
        "postCount" : 3574,
        "uniqueIPCount" : 16,
        "nextNsEmbedRule" : 7,
        "nextPollId" : 1
}

<baris>

It looks like I was right and your database didn't have the global object. It shows nextUid as 1 which isn't correct. Do you have a back up of your database before upgrade? You can restore it into another database and check the correct global object.

TaLoche

@baris you're the man! Thanks so much for taking the time and your help.

This is the result on my snapsnot for my fresh install of nodebb v1.10.1 and with only the mongorestore on it.
Not better, I'm afraid...

> db.objects.findOne({_key: "global"});
{
        "_id" : ObjectId("5dd298e8eb7500a3dd086e72"),
        "_key" : "global",
        "nextEid" : 84,
        "nextCid" : 4,
        "nextUid" : 1,
        "userCount" : 1,
        "nextTid" : 1,
        "topicCount" : 0,
        "nextPid" : 1,
        "postCount" : -1,
        "uniqueIPCount" : 11
}

PitaJ

Did you mongorestore to the same database or a different one?

TaLoche

@PitaJ Hi PitaJ, I'm afraid that I don't have anymore the original Nodebb setup. Before I was centOS and Mongo and with a nodeBB on v1.10.1.
Now I started a new droplet on DigitalOcean with Ubuntu and Mongo. I only have my Mongo dump left for my old setup.
So to answer your question this is a mongorestore in a new database.

PitaJ

Also, did you run ./nodebb setup before or after the restore?

I think what baris was saying is to setup a fresh nodebb with database foo, mongorestore to database nodebb, and then change the DB name in config.json to point to the latter

TaLoche

@PitaJ ok I see what you mean.
I followed this https://www.rosehosting.com/blog/how-to-install-nodebb-on-ubuntu-18-04/
I created a new database on this new setup when everything was up and running, I did the mongorestore into that new database then did a ./nodebb setup and then proceed with the update to v1.12.2.
I tried to set up a new nodeBB with a regular database, dump the old one into another database and do the switch in the config.json then.

TaLoche

@PitaJ Did a new setup with database1, created database 2, did a mongorestore on database 2, changed the config.json to switch to database2 and now I'm trying to run ./nodebb setup but I get the following message Index with name: _key_1_value_-1 already exists with different options


 MongoError: Index with name: _key_1_value_-1 already exists with different options
    at /var/www/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/pool.js:598:61
    at authenticateStragglers (/var/www/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/pool.js:516:16)
    at Connection.messageHandler (/var/www/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/pool.js:552:5)
    at emitMessageHandler (/var/www/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:309:10)
    at Socket.<anonymous> (/var/www/nodebb/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:452:17)
    at Socket.emit (events.js:198:13)
    at addChunk (_stream_readable.js:287:12)
    at readableAddChunk (_stream_readable.js:268:11)
    at Socket.Readable.push (_stream_readable.js:223:10)
    at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)

PitaJ

No you don't need to run ./nodebb setup more than once. Just switch to database 2 and restart.

TaLoche

Thanks a lot @PitaJ and @baris for your help. Everything looks fine.