3.3.0 Upgrade Support

<baris>

@sweetp What do you get when you run this in mongodb cli db.objects.count({_key: "messages:mid"});

sweetp

@baris Im using Postgres for my db

<baris>

Can you modify this file https://github.com/NodeBB/NodeBB/blob/master/src/upgrades/3.3.0/chat_message_mids.js#L43-L44 at that line and change it to

console.log('1');
const count = await db.sortedSetCard(`messages:mid`);
console.log('2');
await db.setObjectField('global', 'messageCount', count);
console.log('3');

Then run the upgrade again with ./nodebb upgrade chat_message_mids let me know what output you get. Seems like that upgrade script is getting stuck.

<baris>

When I test on psql I get the following output, obviously I don't have the same data as you.

Updating NodeBB...

Parsing upgrade scripts...
OK | 1 script(s) found
  → [2023/7/27] Set mid on message objects and create messages:mid...
    [##############] (2/2) 100% 1
2
3
 OK (0.04 seconds)
Schema update complete!

sweetp

@baris I get this:

$ ./nodebb upgrade chat_message_mids

Updating NodeBB...

Parsing upgrade scripts...
OK | 1 script(s) found
  → [2023/7/27] Set mid on message objects and create messages:mid...
    [##############] (15/15) 100%

sweetp

@baris
I replaced L17 - L18 with this:

await batch.processSortedSet(`chat:room:${roomId}:mids`, async (mids) => {
console.log(`mids: ${mids}`);
let messageData = await db.getObjects(mids.map(mid => `message:${mid}`));
console.log(`messageData: ${messageData}`);

and get this:

Updating NodeBB...

Parsing upgrade scripts...
OK | 1 script(s) found
  → [2023/7/27] Set mid on message objects and create messages:mid...
    [##############] (15/15) 100% mids: 24,25,26,27,28,29
mids: 7,8,9,10,22,23
mids: 30,31,32
mids: 33,34,35,36,37,38,39
mids: 15
mids: 11,12,13,14,16,17,18,19
mids: 4,6,20,21
mids: 1,2
mids: 3,5

looks like messageData is never resolving...

<baris>

So it seems it gets stuck in getObjects call can you put the console log in that function.

<baris>

@sweetp I might have found the issue, I will take a look today and publish a patch if necessary.

<baris>

@sweetp Actually can you modify the getObjects method here https://github.com/NodeBB/NodeBB/blob/master/src/database/postgres/hash.js#L131-L153 and let me know the output.

module.getObjects = async function (keys, fields = []) {
	if (!Array.isArray(keys) || !keys.length) {
		return [];
	}
	console.log('a', keys.length, fields.length);
	if (fields.length) {
		return await module.getObjectsFields(keys, fields);
	}
	console.log('b');
	const res = await module.pool.query({
		name: 'getObjects',
		text: `
SELECT h."data"
  FROM UNNEST($1::TEXT[]) WITH ORDINALITY k("_key", i)
  LEFT OUTER JOIN "legacy_object_live" o
               ON o."_key" = k."_key"
  LEFT OUTER JOIN "legacy_hash" h
               ON o."_key" = h."_key"
              AND o."type" = h."type"
 ORDER BY k.i ASC`,
			values: [keys],
	});
	console.log('c');
	return res.rows.map(row => row.data);
};

Let me know what you see after you run the upgrade again with those modifications.

sweetp

@baris I get this:

Updating NodeBB...

Parsing upgrade scripts...
OK | 1 script(s) found
  → [2023/7/27] Set mid on message objects and create messages:mid...
    [##############] (15/15) 100% a 6 0
b
a 2 0
b
a 3 0
b
a 2 0
b
a 4 0
b
a 8 0
b
a 6 0
b
a 7 0
b
a 1 0
b

<baris>

@sweetp not sure why it is getting stuck on the query, did you check psql database for indices? Maybe it is just running very slow?

When I run it on psql 12, I get the result as expected.

Parsing upgrade scripts...
OK | 1 script(s) found
  → [2023/7/27] Set mid on message objects and create messages:mid...
    [##############] (2/2) 100% a 2 0
b
a 4 0
b
c [
  {
    ip: '127.0.0.1',
    mid: 1,
    roomId: '1',
    content: 'this is first room',
    fromuid: 1,
    timestamp: 1692247752077
  },
  {
    ip: '127.0.0.1',
    mid: 2,
    roomId: '1',
    content: 'second message',
    fromuid: 1,
    timestamp: 1692247754008
  }
]
c [
  {
    ip: '127.0.0.1',
    mid: 3,
    roomId: '2',
    content: 'support channgel here',
    fromuid: 1,
    timestamp: 1692247893960
  },
  {
    ip: '127.0.0.1',
    mid: 4,
    roomId: '2',
    content: 'hskdjfhskd',
    fromuid: 1,
    timestamp: 1692247895075
  },
  {
    ip: '127.0.0.1',
    mid: 5,
    roomId: '2',
    content: 'djfhgksdf',
    fromuid: 1,
    timestamp: 1692247895756
  },
  {
    ip: '127.0.0.1',
    mid: 6,
    roomId: '2',
    content: 'jdhgkjdsgd',
    fromuid: 1,
    timestamp: 1692247896428
  }
]
1
2
3
 OK (0.05 seconds)
Schema update complete!

sweetp

I restored my database and forum to v3.2.3 and tried to upgrade again.
but I get the same issue

<baris>

@sweetp when you are on 3.2.3, can you add the same console.logs into getObjects and then load the chat room? Does it load then?

sweetp

@barus, sure thing,
Am I adding the logs to the hash.js file

<baris>

Yeah getObjects function in hash.js for postgress.

sweetp

here's a selection:

<baris>

@sweet seems like the same query is working on 3.2.3, I will try to reproduce. Maybe related to some change in psql dependencies. What version of postgres are you using?

sweetp

@baris 15.4

<baris>

Can you also post the output of the below 2 commands on your working install 3.2.3 and then on 3.3.0 when the upgrade script is freezing.

npm ls pg
npm ls pg-cursor

sweetp

$ npm ls pg
[email protected] /var/www/vhosts/…/NodeBB
├─┬ [email protected]
│ └── [email protected] deduped
├─┬ [email protected]
│ └── [email protected] deduped
└─┬ [email protected]
  └─┬ [email protected]
    └── [email protected] deduped

$ npm ls pg-cursor
[email protected] /var/www/vhosts/…/NodeBB
└── [email protected]

$ npm ls pg
[email protected] /var/www/vhosts/…/NodeBB
├─┬ [email protected]
│ └── [email protected] deduped
├─┬ [email protected]
│ └── [email protected] deduped
└─┬ [email protected]
  └─┬ [email protected]
    └── [email protected] deduped

$ npm ls pg-cursor
[email protected] /var/www/vhosts/…/NodeBB
└── [email protected]