I was looking into trying to figure out server costs with an estimated number of people, but figured it would be best to get some real world data to compare it to! So if you don't mind I would love to hear about your server specs/setup and how many users that handles!
If I understand you correct, you don't want to convert your database, but want NodeBB to work with users, that still will be in your existing database (MS-SQL).
It's possible, by creating your own database driver for NodeBB.
But, actually, it's not so easy. If you don't have high skill in programming, it's near to impossible.
has anyone run NodeBB using a custom SQL user database like this?
Maybe, but I never heard about it. You could try to wait for the NodeBB core developers. They, probably, know something, that may helps you.