If I understand you correct, you don't want to convert your database, but want NodeBB to work with users, that still will be in your existing database (MS-SQL).
It's possible, by creating your own database driver for NodeBB.
But, actually, it's not so easy. If you don't have high skill in programming, it's near to impossible.
has anyone run NodeBB using a custom SQL user database like this?
Maybe, but I never heard about it. You could try to wait for the NodeBB core developers. They, probably, know something, that may helps you.
Yeah I don't really find heroku viable at all, apart from the fact that the workflow is a total pain (unless I'm missing something)... the cost is just too much. What's your "cheap" VPS? Fancy a VPS hosted on SSDs for just $5 a month? http://burnaftercompiling.com/nodebb/setting-up-a-nodebb-forum-for-dummies/