3.3.0 Upgrade Support

[陈洁]

@baris Now it runs successfully. I don’t know why fetch can’t run. On my website

[baris]

@陈洁 What is the value of url in your config.json? Try removing the www if you have it there.

[baris]

@陈洁 Looks like you modified the code and added try/catch around the fetch which is causing the res variable to be undefined further down the code.

It should be

const res = await fetch(url, options);
const { headers } = res;
const contentType = headers.get('content-type');
const isJSON = contentType && contentType.startsWith('application/json');

let response;
if (isJSON) {
	response = await res.json();
} else {
	response = await res.text();
}

if (!res.ok) {
	return cb(new Error(isJSON ? response.status.message : response));
}

[陈洁]

@baris I removed the www. I debugged many times and the fetch was 404. I don’t understand why this happened. I have switched back to the version of $.ajax

[julian]

@陈洁 What browser are you using... have you tried Firefox, etc?

[陈洁]

This post is deleted!

[sweetp]

@baris can I run this migration directly on the database?

[陈洁]

@baris

Using this method solves the problem, but it doesn't feel elegant enough

        let response;
	try {
		if (isJSON) {
			response = await res.json();
		} else {
			response = await res.text();
		}
	} catch (e) {
		response = await res;
	}

[baris]

@sweetp Yes but it would take some time to run the required queries in psql by hand.

I doubt nodebb would work after that though if db.getObjects is failing. You can try running a custom script to see if it fails for everything or just for these message objects.

Place the below code in your nodebb folder and run it with node custom_script.js

/* eslint-disable no-await-in-loop */
/* globals require, console, process */

'use strict';

const nconf = require('nconf');

nconf.file({
	file: 'config.json',
});

nconf.defaults({
	base_dir: __dirname,
	views_dir: './build/public/templates',
	upload_path: 'public/uploads',
});

const db = require('./src/database');

db.init(async (err) => {
	if (err) {
		console.log(`NodeBB could not connect to your database. Error: ${err.message}`);
		process.exit();
	}

	await testDb();
	console.log('done');
	process.exit();
});


async function testDB() {
	const globalObj = await db.getObject('global');
	console.log('global', globalObj);
 	const msgs = await db.getObjects(['message:1', 'message:2', 'message:3']); // you can change the keys to the actual ones from the upgrade script
	console.log(msgs);
}

Try running this script on 3.2.3 and 3.3.0 and see if it fails on 3.3.0 if it does it's time to use git-bisect to find which commit caused the issue.

[baris]

@陈洁 On the register page when you enter a username it makes these requests to check if the username exists and if its available the call returns 404 status code when that happens the contentType is set to "text/plain; charset=utf-8" and isJSON is set to false so it executes response = await res.text()

Is this not the case on your forum?

[sweetp]

@baris said in 3.3.0 Upgrade Support:

node custom_script.js

this works just fine on v3.3.0...

node custom_script.js
global {
  nextCid: 29,
  nextEid: 2070,
  nextMid: 59,
  nextPid: 4177,
  nextTid: 927,
  nextUid: 1482,
  postCount: 4094,
  userCount: 1350,
  loginCount: 1058,
  nextFlagId: 2,
  topicCount: 872,
  uniqueIPCount: 137434,
  nextChatRoomId: 16,
  nextTopicEventId: 24
}
[
  {
    ip: '45.18.15.217',
    roomId: 3,
    content: 'In cookie 5 it says to go to system preference/security & privacy/ and click on full disk access and the check the box next to cookies. I am using Mojave current edition but when I go there it has smbd and sushi but nothing about cookie5. can you help. I downloaded this from the App Store.',
    deleted: 0,
    fromuid: 505,
    timestamp: 1546640245770
  },
  {
    ip: '88.202.177.201',
    roomId: 3,
    content: 'Hi Frank, if you open up you Applications folder. you can drag the Cookie 5 icon into that System Preferences window',
    deleted: 0,
    fromuid: 2,
    timestamp: 1546648541667
  },
  {
    ip: '144.2.74.202',
    roomId: 5,
    content: 'Ich bekomme immer wieder Aufforderungen von Adobe, meinen Flashplayer upzudaten, obwohl ich die neueste Version installiert habe. Ich vermute, dass hat mit Cookie zu tun. Was muss einstellen, damit Adobe Ruhe gibt?',
    deleted: 0,
    fromuid: 412,
    timestamp: 1560601654538
  }
]
done

[baris]

@sweetp can you change the value 500 to 2 here https://github.com/NodeBB/NodeBB/blob/master/src/upgrades/3.3.0/chat_message_mids.js#L41 and try running ./nodebb upgrade chat_message_mids again?

[sweetp]

@baris that works!

[baris]

@sweetp Yeah I think the issue is related to the number of parallel connections that are being opened, I will make some changes to handle this in 3.3.1.

[sweetp]

@baris thanks for all your help

[baris]

@sweetp no worries, thanks for debugging

[fgallese]

@baris said in 3.3.0 Upgrade Support:

You could try adding .allowDiskUse(true) to the query at https://github.com/NodeBB/NodeBB/blob/master/src/database/mongo/sorted.js#L397-L401.

It throws that error because the data is too big to sort in memory (the default limit is 100mb).

I am working on the new upgrade script that should be aviable later today as 3.3.1.

Hello @baris . Today after update to 3.3.1 I see a new error:

Parsing upgrade scripts... 
OK | 5 script(s) found, 124 skipped
  → [2023/7/2] Update chat messages to add roomId field...
Error occurred
2023-08-18T19:43:48.782Z [4567/9] - warn: NodeBB Setup Aborted.
 TypeError: midsSeen.hasOwnProperty is not a function
    at /usr/src/app/nodebb/src/upgrades/3.3.0/chat_room_refactor.js:41:55
    at Array.filter (<anonymous>)
    at batch.processSortedSet.batch (/usr/src/app/nodebb/src/upgrades/3.3.0/chat_room_refactor.js:41:31)
    at module.processSortedSet (/usr/src/app/nodebb/src/database/mongo/sorted.js:588:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async exports.processSortedSet (/usr/src/app/nodebb/src/batch.js:30:10)
    at async batch.processSortedSet.batch (/usr/src/app/nodebb/src/upgrades/3.3.0/chat_room_refactor.js:40:8)
    at async module.processSortedSet (/usr/src/app/nodebb/src/database/mongo/sorted.js:588:5)
    at async exports.processSortedSet (/usr/src/app/nodebb/src/batch.js:30:10)
    at async batch.processArray.batch (/usr/src/app/nodebb/src/upgrades/3.3.0/chat_room_refactor.js:38:6)

Any idea how can I fix this ?

Thank you

[baris]

@fgallese change this line https://github.com/NodeBB/NodeBB/blob/master/src/upgrades/3.3.0/chat_room_refactor.js#L36 to const midsSeen = {};

[baris]

@fgallese released 3.3.2 to address this issue.

[fgallese]

@baris I am still unable to do the upgrade.. the mongo connection goes down after several minutes running the update:

nodebb  | Parsing upgrade scripts... 
nodebb  | OK | 5 script(s) found, 124 skipped
nodebb  |   → [2023/7/2] Update chat messages to add roomId field...
nodebb  | Error occurred
    [###          ] (502/1775) 28% 2023-08-19T18:27:29.351Z [4567/10] - warn: NodeBB Setup Aborted.
nodebb  |  MongoServerError: cursor id 8839582387884439579 not found
nodebb  |     at Connection.onMessage (/usr/src/app/nodebb/node_modules/mongodb/lib/cmap/connection.js:202:26)
nodebb  |     at MessageStream.<anonymous> (/usr/src/app/nodebb/node_modules/mongodb/lib/cmap/connection.js:61:60)
nodebb  |     at MessageStream.emit (node:events:513:28)
nodebb  |     at processIncomingData (/usr/src/app/nodebb/node_modules/mongodb/lib/cmap/message_stream.js:124:16)
nodebb  |     at MessageStream._write (/usr/src/app/nodebb/node_modules/mongodb/lib/cmap/message_stream.js:33:9)
nodebb  |     at writeOrBuffer (node:internal/streams/writable:392:12)
nodebb  |     at _write (node:internal/streams/writable:333:10)
nodebb  |     at Writable.write (node:internal/streams/writable:337:10)
nodebb  |     at Socket.ondata (node:internal/streams/readable:766:22)
nodebb  |     at Socket.emit (node:events:513:28)
nodebb exited with code 1

I guess my only option is to remove chat messages. Is there any tool for this or am I gonna have to figure it out in the DB ? Can I just run a delete on {_key:"message:*"} where the timestamp is older than 1 month or something ?

If I run this, would i break something ?

db.objects.deleteMany({
  "_key": {
    "$regex": /^message:\d+$/
  },
  "timestamp": { "$lt": oneMonthAgoTimestamp }
});