If I understand you correct, you don't want to convert your database, but want NodeBB to work with users, that still will be in your existing database (MS-SQL).
It's possible, by creating your own database driver for NodeBB.
But, actually, it's not so easy. If you don't have high skill in programming, it's near to impossible.
has anyone run NodeBB using a custom SQL user database like this?
Maybe, but I never heard about it. You could try to wait for the NodeBB core developers. They, probably, know something, that may helps you.