All the API token erased from the admin page

[nullpointer]

I have a bit serious issue in my production environment, where the API token (including master token) periodically erased from the admin page. The client I use to connect to the NodeBB API is unable to connect anymore since the API is erased. This incident is not happening in development environment since the number of API tokens generated is quite low. The api token in production environment can reach up to thousands of API tokens.

Is there any limit or restriction related to the size of the generated API token? Like the number of the API generated or the frequency of the token used by another http client?

Thanks in advance.

[julian]

@nullpointer I don't know. Likely 16mb but likely much lower since the tokens are stored in the form of JSON, and not as a properly hashed entry in the database.

It sounds like you might be running into this issue, which is causing a catastrophic loss of the entire key set if you run out of space? Interesting.

Well, interesting for me, much less so for you.

---

I need to refactor this system anyway. It's implemented in a way that I do not like. It will be done for v3.1.0, which is likely due several weeks after v3.0.0.

[nullpointer]

@julian thanks for your quick reply

Could you suggest me any work around for this? Maybe I can modify the existing code for a hot fix?

Also could you point me out the code that store the data in a json file?

Thanks a lot

[julian]

https://github.com/NodeBB/NodeBB/issues/11520

Unfortunately it's not something I can hotfix (otherwise I would have definitely fixed it for v3).

The entire system needs a refactor, and the only thing I can offer now is that you limit the number of tokens you make in the meantime.

I will try my best to ensure that the refactor is backportable to v2.x

[baris]

@nullpointer how are you creating these tokens? What database are you using? Are the tokens still in the database when you look at the keys settings:core.api:sorted-list:tokens:0. Each sorted-list item is stored as a separate document so it shouldn't hit the limit I think.

[julian]

@baris brings up a good point, I was mistaken about what I thought was a limitation of the sorted-list library.

Are you certain the tokens are gone (as in, requests using them no longer work)?

[nullpointer]

@baris I use the existing code:

        const settings = await meta.settings.get('core.api');
	settings.tokens = settings.tokens || [];

	const newToken = {
		token: utils.generateUUID(),
		uid: req.params.uid,
		description: req.body.description || '',
		timestamp: Date.now(),
	};
	settings.tokens.push(newToken);
	await meta.settings.set('core.api', settings);

@baris said in All the API token erased from the admin page:

Are the tokens still in the database when you look at the keys settings:core.api:sorted-list:tokens:0

I don't think so. The tokens is completely gone in DB. Here is the result of the db query

Here is the result using regex with prefix ^setting:core.api

[nullpointer]

@julian yes, I am sure. the admin token and the user tokens are all gone both in admin page and in the database

[baris]

Is there more code after that? When I try the same code it doesn't delete the existing keys from the database.

[baris]

The only way those items would be emptied is if you call meta.settings.set with an empty array of tokens.

const settings = await meta.settings.get('core.api');
settings.tokens = [];
await meta.settings.set('core.api', settings);

Maybe put a console.log inside meta.settings.set and see if it is being called like that.

[nullpointer]

@baris there is only one line of code after that:

helpers.formatApiResponse(200, res, newToken);

@baris said in All the API token erased from the admin page:

Maybe put a console.log inside meta.settings.set and see if it is being called like that.

I will try to put this logging and see the behavior

[julian]

Yes, that's an unfortunate downside of the sorted-list system we use. It's easy to accidentally overwrite everything like that, which in hindsight, means I should not have used sorted-list for it

[nullpointer]

@julian is there any workaround for this particular issue? Currently I need to disable the forum feature in my production environment

[nullpointer]

Hello @julian, could you give me some references regarding this sorted-list library that you mention?

[nullpointer]

Solved after upgrading to v3.1.4
Reference: https://github.com/NodeBB/NodeBB/pull/11533/commits

Thanks @julian for addressing this issue.

[julian]

@nullpointer glad to hear it