SendGrid Contact List Sync Crashing NodeBB

tankerkiller125

If I click on "Synchronize" in the admin dashboard I get the following error:

2023-07-16T21:57:44.107Z [4567/3526933] - error: uncaughtException: Cannot set headers after they are sent to the client
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
    at new NodeError (node:internal/errors:387:5)
    at ServerResponse.setHeader (node:_http_outgoing:644:11)
    at ServerResponse.header (/root/sysadminszone/nodebb/node_modules/express/lib/response.js:794:10)
    at ServerResponse.contentType (/root/sysadminszone/nodebb/node_modules/express/lib/response.js:624:15)
    at ServerResponse.sendStatus (/root/sysadminszone/nodebb/node_modules/express/lib/response.js:373:8)
    at Emailer.marketing.synchronize (/root/sysadminszone/nodebb/node_modules/nodebb-plugin-emailer-sendgrid/index.js:429:6) {"date":"Sun Jul 16 2023 21:57:44 GMT+0000 (Coordinated Universal Time)","error":{"code":"ERR_HTTP_HEADERS_SENT"},"exception":true,"os":{"loadavg":[0.07,0.13,0.16],"uptime":3913745.08},"process":{"argv":["/usr/bin/node","/root/sysadminszone/nodebb/app.js"],"cwd":"/root/sysadminszone/nodebb","execPath":"/usr/bin/node","gid":0,"memoryUsage":{"arrayBuffers":19755039,"external":23292956,"heapTotal":122163200,"heapUsed":108030520,"rss":175153152},"pid":3526933,"uid":0,"version":"v16.20.1"},"stack":"Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client\n    at new NodeError (node:internal/errors:387:5)\n    at ServerResponse.setHeader (node:_http_outgoing:644:11)\n    at ServerResponse.header (/root/sysadminszone/nodebb/node_modules/express/lib/response.js:794:10)\n    at ServerResponse.contentType (/root/sysadminszone/nodebb/node_modules/express/lib/response.js:624:15)\n    at ServerResponse.sendStatus (/root/sysadminszone/nodebb/node_modules/express/lib/response.js:373:8)\n    at Emailer.marketing.synchronize (/root/sysadminszone/nodebb/node_modules/nodebb-plugin-emailer-sendgrid/index.js:429:6)","trace":[{"column":5,"file":"node:internal/errors","function":"new NodeError","line":387,"method":null,"native":false},{"column":11,"file":"node:_http_outgoing","function":"ServerResponse.setHeader","line":644,"method":"setHeader","native":false},{"column":10,"file":"/root/sysadminszone/nodebb/node_modules/express/lib/response.js","function":"ServerResponse.header","line":794,"method":"header","native":false},{"column":15,"file":"/root/sysadminszone/nodebb/node_modules/express/lib/response.js","function":"ServerResponse.contentType","line":624,"method":"contentType","native":false},{"column":8,"file":"/root/sysadminszone/nodebb/node_modules/express/lib/response.js","function":"ServerResponse.sendStatus","line":373,"method":"sendStatus","native":false},{"column":6,"file":"/root/sysadminszone/nodebb/node_modules/nodebb-plugin-emailer-sendgrid/index.js","function":"Emailer.marketing.synchronize","line":429,"method":"synchronize","native":false}]}
2023-07-16T21:57:44.107Z [4567/3526933] - error: Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
    at new NodeError (node:internal/errors:387:5)
    at ServerResponse.setHeader (node:_http_outgoing:644:11)
    at ServerResponse.header (/root/sysadminszone/nodebb/node_modules/express/lib/response.js:794:10)
    at ServerResponse.contentType (/root/sysadminszone/nodebb/node_modules/express/lib/response.js:624:15)
    at ServerResponse.sendStatus (/root/sysadminszone/nodebb/node_modules/express/lib/response.js:373:8)
    at Emailer.marketing.synchronize (/root/sysadminszone/nodebb/node_modules/nodebb-plugin-emailer-sendgrid/index.js:429:6)
2023-07-16T21:57:44.108Z [4567/3526933] - info: [app] Shutdown (SIGTERM/SIGINT) Initialised.
2023-07-16T21:57:44.110Z [4567/3526933] - info: [app] Web server closed to connections.
2023-07-16T21:57:44.114Z [4567/3526933] - info: [app] Live analytics saved.
2023-07-16T21:57:44.125Z [4567/3526933] - info: [app] Database connection closed.
2023-07-16T21:57:44.125Z [4567/3526933] - info: [app] Shutdown complete.

We use the contact list for things like sending ToS updates and stuff, so it's important that the list in SendGrid is accurate and the same as the forum itself.

Any assistance on getting this fixed would be greatly appreciated.

<baris>

@tankerkiller125 try with 3.1.2, https://github.com/julianlam/nodebb-plugin-emailer-sendgrid/commit/ef7f2fbcef31e6eaad94a42f113ead2393939894.

There might be another when you sync but you shouldn't get the Cannot set headers after they are sent to the client error anymore.

tankerkiller125

Oh, I forgot the plugin information, that's here:

Active plugins:
        ....
        * [email protected] (installed, enabled)
        ....

<baris>

@tankerkiller125 try with 3.1.2, https://github.com/julianlam/nodebb-plugin-emailer-sendgrid/commit/ef7f2fbcef31e6eaad94a42f113ead2393939894.

There might be another when you sync but you shouldn't get the Cannot set headers after they are sent to the client error anymore.

tankerkiller125

@baris Well it's not longer crashing, but I'm just getting:

2023-07-17T00:54:27.021Z [4568/3628540] - info: [plugins/emailer-sendgrid] Synchronizing...
2023-07-17T00:54:27.695Z [4568/3628540] - warn: [plugins/emailer-sendgrid] Unable to synchronize

I've triple checked that the SendGrid API key has access to both the Mail API and Marketing API so that's not causing the issue. Unfortunately that's all the information it's currently giving me.

<baris>

@tankerkiller125 try with 3.1.3, it should show the error message from sendgrid.

tankerkiller125

Crashing again:

2023-07-17T01:53:21.480Z [4568/3629500] - info: [plugins/emailer-sendgrid] Synchronizing...
2023-07-17T01:53:22.284Z [4568/3629500] - error: uncaughtException: err is not defined
ReferenceError: err is not defined
    at Emailer.marketing.synchronize (/root/sysadminszone/nodebb/node_modules/nodebb-plugin-emailer-sendgrid/index.js:424:70)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {"date":"Mon Jul 17 2023 01:53:22 GMT+0000 (Coordinated Universal Time)","error":{},"exception":true,"os":{"loadavg":[1.61,0.56,0.25],"uptime":3927883.26},"process":{"argv":["/usr/bin/node","/root/sysadminszone/nodebb/app.js"],"cwd":"/root/sysadminszone/nodebb","execPath":"/usr/bin/node","gid":0,"memoryUsage":{"arrayBuffers":19707881,"external":27309238,"heapTotal":101437440,"heapUsed":97141448,"rss":171921408},"pid":3629500,"uid":0,"version":"v16.20.1"},"stack":"ReferenceError: err is not defined\n    at Emailer.marketing.synchronize (/root/sysadminszone/nodebb/node_modules/nodebb-plugin-emailer-sendgrid/index.js:424:70)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)","trace":[{"column":70,"file":"/root/sysadminszone/nodebb/node_modules/nodebb-plugin-emailer-sendgrid/index.js","function":"Emailer.marketing.synchronize","line":424,"method":"synchronize","native":false},{"column":null,"file":null,"function":"runMicrotasks","line":null,"method":null,"native":false},{"column":5,"file":"node:internal/process/task_queues","function":"processTicksAndRejections","line":96,"method":null,"native":false}]}
2023-07-17T01:53:22.285Z [4568/3629500] - error: ReferenceError: err is not defined
    at Emailer.marketing.synchronize (/root/sysadminszone/nodebb/node_modules/nodebb-plugin-emailer-sendgrid/index.js:424:70)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
2023-07-17T01:53:22.286Z [4568/3629500] - info: [app] Shutdown (SIGTERM/SIGINT) Initialised.
2023-07-17T01:53:22.291Z [4568/3629500] - info: [app] Web server closed to connections.
2023-07-17T01:53:22.307Z [4568/3629500] - info: [app] Live analytics saved.

<baris>

@tankerkiller125 try with 3.1.4.

tankerkiller125

@baris Not much extra information I'm afraid:

2023-07-17T02:14:53.808Z [4568/3630129] - warn: [plugins/emailer-sendgrid] Unable to synchronize.
Error: Bad Request
    at node_modules/@sendgrid/client/src/classes/client.js:146:29
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

I wonder if maybe SendGrid changed something in their API that's breaking things?

<baris>

Maybe, this is the request the plugin is making.

await Client.request({
	method: 'PUT',
	url: `/v3/marketing/contacts`,
	body: {
		list_ids: [Emailer._settings['marketing.id']],
		contacts: data.map((entry) => {
			const fields = ['username', 'fullname'].reduce((memo, prop) => {
				memo[Emailer._settings[`marketing.fields.${prop}`]] = entry[prop] || '';
				return memo;
			}, {});
			return {
				email: entry.email,
				custom_fields: fields,
			};
		}),
	},
});

Do you have marketting.id set in plugin settings? Maybe add some console.logs in the plugin to verify if all the data is passed in.

tankerkiller125

@baris Where would I set that? I don't see any options for it in the emailer plugin settings in ACP nor in the Email settings.

I will note though that when I click on "View list in SendGrid" it does take me to the correct list.

<baris>

You can add it right after this line https://github.com/julianlam/nodebb-plugin-emailer-sendgrid/blob/master/index.js#L399 console.log(Emailer._settings['marketing.id']) restart nodebb and click sync again to see what it prints.

tankerkiller125

@baris I think I may have found the issue, I ended up console logging the data.map function to console.log({email: entry.email, custom_fields: fields}) and what I discovered is that somehow, some way, a couple of users exist that have zero email associated with them. Which would in fact cause the 400 error I'm experiencing.

Deleting those user accounts (no content or posts) resolved the issue.

<baris>

@tankerkiller125 could be related to the changes made in core, email is optional now so there could be users with no email. We should skip those in the plugin if it causes an issue with the api call.

tankerkiller125

@baris If no email, user accounts are allowed now yeah skipping them would be required otherwise the SendGrid API breaks.

<baris>

@tankerkiller125 this is done in 3.1.5 https://github.com/julianlam/nodebb-plugin-emailer-sendgrid/commit/dd834cbf47a053644c3ebfb67e799a6fd9a929e9