In general, we recommend MongoDB if you have a large community. Redis will work just fine, although the memory usage does scale up over time as everything is stored in-memory. Just something to consider.
If I understand you correct, you don't want to convert your database, but want NodeBB to work with users, that still will be in your existing database (MS-SQL).
It's possible, by creating your own database driver for NodeBB.
But, actually, it's not so easy. If you don't have high skill in programming, it's near to impossible.
has anyone run NodeBB using a custom SQL user database like this?
Maybe, but I never heard about it. You could try to wait for the NodeBB core developers. They, probably, know something, that may helps you.