3.3.0 Upgrade Support

<baris>

I was on 18.x, tested on 20.5.1 works there too, although I found this which sounds similar https://github.com/brianc/node-postgres/issues/2069

fgallese

I am getting this error when upgrading:

Parsing upgrade scripts... 
OK | 5 script(s) found, 124 skipped
  → [2023/7/2] Update chat messages to add roomId field...
Error occurred
    [##############] (401/401) 100% 2023-08-17T20:42:31.417Z [4567/9] - warn: NodeBB Setup Aborted.
 MongoServerError: error while multiplanner was selecting best plan :: caused by :: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting.
    at Connection.onMessage (/usr/src/app/nodebb/node_modules/mongodb/lib/cmap/connection.js:202:26)
    at MessageStream.<anonymous> (/usr/src/app/nodebb/node_modules/mongodb/lib/cmap/connection.js:61:60)
    at MessageStream.emit (node:events:513:28)
    at processIncomingData (/usr/src/app/nodebb/node_modules/mongodb/lib/cmap/message_stream.js:124:16)
    at MessageStream._write (/usr/src/app/nodebb/node_modules/mongodb/lib/cmap/message_stream.js:33:9)
    at writeOrBuffer (node:internal/streams/writable:392:12)
    at _write (node:internal/streams/writable:333:10)
    at Writable.write (node:internal/streams/writable:337:10)
    at Socket.ondata (node:internal/streams/readable:766:22)
    at Socket.emit (node:events:513:28)

Anyone else seen this issue ?

<baris>

@fgallese do you have a room with a lot of users by any chance? Or were you using nodebb-plugin-global-chat?

fgallese

@baris said in NodeBB 3.3.0:

@fgallese do you have a room with a lot of users by any chance? Or were you using nodebb-plugin-global-chat?

Yes and yes. We have a chat room with all users which sees a lot of usage, so it is expected for us to have a lot of messages in the history I guess.

<baris>

Hmm in that case I will have to modify that upgrade script so it doesn't load all uids/messageIds in the room.

fgallese

@baris said in NodeBB 3.3.0:

Hmm in that case I will have to modify that upgrade script so it doesn't load all uids/messageIds in the room.

Do you think there is any quick workaround I can apply in my db to help this ? Like message trimming or similar ?

<baris>

You could try adding .allowDiskUse(true) to the query at https://github.com/NodeBB/NodeBB/blob/master/src/database/mongo/sorted.js#L397-L401.

It throws that error because the data is too big to sort in memory (the default limit is 100mb).

I am working on the new upgrade script that should be aviable later today as 3.3.1.

陈洁

Sign up button doesn't work,Will not jump after registration v3.3.0

<baris>

@fgallese can you try replacing src/upgrades/3.3.0/chat_room_refactor.js with the new one from https://github.com/NodeBB/NodeBB/blob/master/src/upgrades/3.3.0/chat_room_refactor.js.

After that try running it again, it will run slower but shouldn't cause the memory limit issue.

<baris>

@陈洁 Can you reproduce on this forum? We are running 3.3.0 here as well.

陈洁

@baris It can be successful here. On my site, a breakpoint error is reported in fetch, which seems to be the reason for packaging.

I replaced fetch with $.ajax and restored

async function xhr(options, cb) {
	// Allow options to be modified by plugins, etc.
	({ options } = await fireHook('filter:api.options', { options }));

	$.ajax(options)
		.done((res) => {
			cb(null, (
				res &&
				res.hasOwnProperty('status') &&
				res.hasOwnProperty('response') ? res.response : (res || {})
			));
		})
		.fail((ev) => {
			let errMessage;
			if (ev.responseJSON) {
				errMessage = ev.responseJSON.status && ev.responseJSON.status.message ?
					ev.responseJSON.status.message :
					ev.responseJSON.error;
			}

			cb(new Error(errMessage || ev.statusText));
		});
}

陈洁

@baris Now it runs successfully. I don’t know why fetch can’t run. On my website

<baris>

@陈洁 What is the value of url in your config.json? Try removing the www if you have it there.

<baris>

@陈洁 Looks like you modified the code and added try/catch around the fetch which is causing the res variable to be undefined further down the code.

It should be

const res = await fetch(url, options);
const { headers } = res;
const contentType = headers.get('content-type');
const isJSON = contentType && contentType.startsWith('application/json');

let response;
if (isJSON) {
	response = await res.json();
} else {
	response = await res.text();
}

if (!res.ok) {
	return cb(new Error(isJSON ? response.status.message : response));
}

陈洁

@baris I removed the www. I debugged many times and the fetch was 404. I don’t understand why this happened. I have switched back to the version of $.ajax

julian

@陈洁 What browser are you using... have you tried Firefox, etc?

陈洁

This post is deleted!

sweetp

@baris can I run this migration directly on the database?

陈洁

@baris

Using this method solves the problem, but it doesn't feel elegant enough

        let response;
	try {
		if (isJSON) {
			response = await res.json();
		} else {
			response = await res.text();
		}
	} catch (e) {
		response = await res;
	}

<baris>

@sweetp Yes but it would take some time to run the required queries in psql by hand.

I doubt nodebb would work after that though if db.getObjects is failing. You can try running a custom script to see if it fails for everything or just for these message objects.

Place the below code in your nodebb folder and run it with node custom_script.js

/* eslint-disable no-await-in-loop */
/* globals require, console, process */

'use strict';

const nconf = require('nconf');

nconf.file({
	file: 'config.json',
});

nconf.defaults({
	base_dir: __dirname,
	views_dir: './build/public/templates',
	upload_path: 'public/uploads',
});

const db = require('./src/database');

db.init(async (err) => {
	if (err) {
		console.log(`NodeBB could not connect to your database. Error: ${err.message}`);
		process.exit();
	}

	await testDb();
	console.log('done');
	process.exit();
});


async function testDB() {
	const globalObj = await db.getObject('global');
	console.log('global', globalObj);
 	const msgs = await db.getObjects(['message:1', 'message:2', 'message:3']); // you can change the keys to the actual ones from the upgrade script
	console.log(msgs);
}

Try running this script on 3.2.3 and 3.3.0 and see if it fails on 3.3.0 if it does it's time to use git-bisect to find which commit caused the issue.