Fixed as of https://github.com/NodeBB/NodeBB/commit/5ec289eee2c44f4dd49e7b2884d27dd1acb20d5c.
The problem was I was deleting tags that no longer had topics but was doing it in the wrong place. If a user couldn't see any topics the code was deleting the tag. I moved it up to actually check the tids and not the topics the user can see.