Bulk "unsubscribe" emails

Dave Jensen

As a few of you might recall, I migrated a forum that was running SMF for about 16 years until I migrated to NodeBB a year ago.

As such, I have a lot of email address that either don't exist or don't want digest emails. I just switched to Postmark and I don't want to ruin my email reputation there when I turn digests back on.

I think there should be some features around making this easier. Webhooks are great but it looks like they're only implemented for the SendGrid plugin.

Feature request:

Upload a CSV of emails to "unsubscribe". This would be for spam complaints and soft bounces.
Upload a CSV of emails to block or invalidate or basically "never send to this email". This would be for hard bounces.

Support request:

When an email is "not validated" what exactly does that mean? What does and does not happen to/with this user?
What's the best way to do each of the above manually?
- I know it will involve updating the DB but I'll need some help with that. The DB schema isn't really documented at all and it's also somewhat non-traditional.

Dave Jensen

I think user email "state" is a bigger problem than I expected. I looked at the Sendgrid mailer code for some clues ...

The webhooks that report back spam complaints, bounces, dropped, etc. don't actually do anything. If somebody reports spam or if an address has a hard bounce, then NodeBB will just keep trying to send and rely on the provider to do something about it. This can get expensive or ruin your email reputation with a provider.

There needs to be more state for user email addresses. Even sites that haven't been around for decades get spam reports and hard bounces. It would be great if I could set an user/email to:

unsubscribed (send no emails except account recovery related)
- Or this could be just the same as auto changing the user settings so that none of the alerts are email and digest is off for the user.
blocked (never send anything to this email)

cc: @baris @julian

Maybe related: https://github.com/NodeBB/NodeBB/issues/8016

julian

Well, to be honest, emails come and go, but I don't want to rely on hard bounces as a method of confirming email validity (it's why we have the email confirmation flow).

Mostly, this is because dealing with emails is very complicated, which is admittedly why we prefer to farm out the dirty work to third-party mailers like SendGrid, et al.

As commented therein, a one-click unsub is a great idea. NodeBB actually does support one-click unsub. We send a unique link in the email header that you can POST.

Unfortunately, SendGrid doesn't pass on the requisite headers, and so GMail strips it. That's an ongoing issue. We'll probably just add that link into the footer.

Dave Jensen

Footer would be great.

I understand the workflow of email confirmation but you're missing the point. Even your community forum hasn't been around very long. And it certainly wasn't migrated from something that was running for 2 decades.

Being able to do this is a must have for email health. Email health is crucial to a forum community. I'm fairly certain the competition understands this.

Farming this out to providers is a problem because it consumes API calls driving up costs. Some providers don't even have a way for me to upload a suppression list. So I have to ruin my reputation with them in order to get the known old addresses in their system.

It's a bad look to not address bad emails with both the system generating (NodeBB) as well as sending (transactional email provider). Heck, what if Stop Forum Spam says an email address is spam, I still have zero way to deal with this. Email has been my number one frustration with NodeBB and continues to be.

I have about 3k email address that I just shouldn't be sending to but I have zero ways to deal with this.

Dave Jensen

The solution doesn't have to be complicated. I just need a way to "never send" email to certain old accounts.

This is the minimum I would like to see:

Unsubscribe all:

One click link on emails (this is being developed as discussed)
Upload a CSV list of emails to unsubscribe all
A webhook to unsubscribe all when received from the transactional email provider.

Dave Jensen

Looks like this line does the unsubscribe work:
https://github.com/NodeBB/NodeBB/blob/19b63bfaa7d14b48991c83ad6dc40f981d61b042/src/controllers/accounts/settings.js#L173

const readline = require('readline');
const fs = require('fs');
const notifications = require('./notifications');

const unsubscribeAll = async function (email) {
    const uid = await user.getUidByEmail(email.toLowerCase());
    if (!uid) return false;
    await user.setSetting(uid, 'dailyDigestFreq', 'off');
    await user.updateDigestSetting(uid, 'off');
    
    for (const type of notifications.baseTypes) {
        const current = await db.getObjectField('user:' + uid + ':settings', 'notificationType_' + type);
        await user.setSetting(uid, 'notificationType_' + type, (current === 'notificationemail' ? 'notification' : 'none'));   
    } 
    return true;
}



const rl = readline.createInterface({
    input: fs.createReadStream('emails.txt') //one email per line
});

rl.on('line', function(email) {
    let success = unsubscribeAll(email);
    console.log(`${email}: ${success ? '✔' : '❌'}`);
});

Notification types:
https://github.com/NodeBB/NodeBB/blob/0037a0388636a92765a3bb16f81dd121ca33e513/src/notifications.js#L20

~~The main thing I don't have are the various values for type. I'm guessing there is an array in the code somewhere.~~

Dave Jensen

@julian As a followup, I received an email from Postmark. They basically told me that they prefer that customers manage their own lists. They're a transnational email provider not a service to maintain lists. I'm with them on this. It's the responsibility of the owner of the list to maintain the list. Right now NodeBB punts that to the email provider.