3.3.0 Upgrade Support

<baris>

@sweetp Actually can you modify the getObjects method here https://github.com/NodeBB/NodeBB/blob/master/src/database/postgres/hash.js#L131-L153 and let me know the output.

module.getObjects = async function (keys, fields = []) {
	if (!Array.isArray(keys) || !keys.length) {
		return [];
	}
	console.log('a', keys.length, fields.length);
	if (fields.length) {
		return await module.getObjectsFields(keys, fields);
	}
	console.log('b');
	const res = await module.pool.query({
		name: 'getObjects',
		text: `
SELECT h."data"
  FROM UNNEST($1::TEXT[]) WITH ORDINALITY k("_key", i)
  LEFT OUTER JOIN "legacy_object_live" o
               ON o."_key" = k."_key"
  LEFT OUTER JOIN "legacy_hash" h
               ON o."_key" = h."_key"
              AND o."type" = h."type"
 ORDER BY k.i ASC`,
			values: [keys],
	});
	console.log('c');
	return res.rows.map(row => row.data);
};

Let me know what you see after you run the upgrade again with those modifications.

sweetp

@baris I get this:

Updating NodeBB...

Parsing upgrade scripts...
OK | 1 script(s) found
  → [2023/7/27] Set mid on message objects and create messages:mid...
    [##############] (15/15) 100% a 6 0
b
a 2 0
b
a 3 0
b
a 2 0
b
a 4 0
b
a 8 0
b
a 6 0
b
a 7 0
b
a 1 0
b

<baris>

@sweetp not sure why it is getting stuck on the query, did you check psql database for indices? Maybe it is just running very slow?

When I run it on psql 12, I get the result as expected.

Parsing upgrade scripts...
OK | 1 script(s) found
  → [2023/7/27] Set mid on message objects and create messages:mid...
    [##############] (2/2) 100% a 2 0
b
a 4 0
b
c [
  {
    ip: '127.0.0.1',
    mid: 1,
    roomId: '1',
    content: 'this is first room',
    fromuid: 1,
    timestamp: 1692247752077
  },
  {
    ip: '127.0.0.1',
    mid: 2,
    roomId: '1',
    content: 'second message',
    fromuid: 1,
    timestamp: 1692247754008
  }
]
c [
  {
    ip: '127.0.0.1',
    mid: 3,
    roomId: '2',
    content: 'support channgel here',
    fromuid: 1,
    timestamp: 1692247893960
  },
  {
    ip: '127.0.0.1',
    mid: 4,
    roomId: '2',
    content: 'hskdjfhskd',
    fromuid: 1,
    timestamp: 1692247895075
  },
  {
    ip: '127.0.0.1',
    mid: 5,
    roomId: '2',
    content: 'djfhgksdf',
    fromuid: 1,
    timestamp: 1692247895756
  },
  {
    ip: '127.0.0.1',
    mid: 6,
    roomId: '2',
    content: 'jdhgkjdsgd',
    fromuid: 1,
    timestamp: 1692247896428
  }
]
1
2
3
 OK (0.05 seconds)
Schema update complete!

sweetp

I restored my database and forum to v3.2.3 and tried to upgrade again.
but I get the same issue

<baris>

@sweetp when you are on 3.2.3, can you add the same console.logs into getObjects and then load the chat room? Does it load then?

sweetp

@barus, sure thing,
Am I adding the logs to the hash.js file

<baris>

Yeah getObjects function in hash.js for postgress.

sweetp

here's a selection:

<baris>

@sweet seems like the same query is working on 3.2.3, I will try to reproduce. Maybe related to some change in psql dependencies. What version of postgres are you using?

sweetp

@baris 15.4

<baris>

Can you also post the output of the below 2 commands on your working install 3.2.3 and then on 3.3.0 when the upgrade script is freezing.

npm ls pg
npm ls pg-cursor

sweetp

$ npm ls pg
[email protected] /var/www/vhosts/…/NodeBB
├─┬ [email protected]
│ └── [email protected] deduped
├─┬ [email protected]
│ └── [email protected] deduped
└─┬ [email protected]
  └─┬ [email protected]
    └── [email protected] deduped

$ npm ls pg-cursor
[email protected] /var/www/vhosts/…/NodeBB
└── [email protected]

$ npm ls pg
[email protected] /var/www/vhosts/…/NodeBB
├─┬ [email protected]
│ └── [email protected] deduped
├─┬ [email protected]
│ └── [email protected] deduped
└─┬ [email protected]
  └─┬ [email protected]
    └── [email protected] deduped

$ npm ls pg-cursor
[email protected] /var/www/vhosts/…/NodeBB
└── [email protected]

<baris>

Just to rule out that it's the dependency upgrade you can try to use the versions from 3.2.3 and try to run the upgrade script.

Install pg & pg-cursor that 3.2.3 uses with the below commands than try to run the upgrade script again.

npm i [email protected]
npm i [email protected]

sweetp

@baris said in 3.3.0 Upgrade Support:

npm i [email protected]

still stalling...

4. Updating NodeBB data store schema...
Parsing upgrade scripts...
OK | 1 script(s) found, 129 skipped
  → [2023/7/27] Set mid on message objects and create messages:mid...
    [##############] (16/16) 100%

<baris>

I will give it a shot with psql 15 and see if I can reproduce that way.

<baris>

@sweetp works for me on psql15 as well. What version of nodejs are you using?

sweetp

@baris Node v20.5.1

<baris>

I was on 18.x, tested on 20.5.1 works there too, although I found this which sounds similar https://github.com/brianc/node-postgres/issues/2069

fgallese

I am getting this error when upgrading:

Parsing upgrade scripts... 
OK | 5 script(s) found, 124 skipped
  → [2023/7/2] Update chat messages to add roomId field...
Error occurred
    [##############] (401/401) 100% 2023-08-17T20:42:31.417Z [4567/9] - warn: NodeBB Setup Aborted.
 MongoServerError: error while multiplanner was selecting best plan :: caused by :: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting.
    at Connection.onMessage (/usr/src/app/nodebb/node_modules/mongodb/lib/cmap/connection.js:202:26)
    at MessageStream.<anonymous> (/usr/src/app/nodebb/node_modules/mongodb/lib/cmap/connection.js:61:60)
    at MessageStream.emit (node:events:513:28)
    at processIncomingData (/usr/src/app/nodebb/node_modules/mongodb/lib/cmap/message_stream.js:124:16)
    at MessageStream._write (/usr/src/app/nodebb/node_modules/mongodb/lib/cmap/message_stream.js:33:9)
    at writeOrBuffer (node:internal/streams/writable:392:12)
    at _write (node:internal/streams/writable:333:10)
    at Writable.write (node:internal/streams/writable:337:10)
    at Socket.ondata (node:internal/streams/readable:766:22)
    at Socket.emit (node:events:513:28)

Anyone else seen this issue ?

<baris>

@fgallese do you have a room with a lot of users by any chance? Or were you using nodebb-plugin-global-chat?