If I understand you correct, you don't want to convert your database, but want NodeBB to work with users, that still will be in your existing database (MS-SQL).
It's possible, by creating your own database driver for NodeBB.
But, actually, it's not so easy. If you don't have high skill in programming, it's near to impossible.
has anyone run NodeBB using a custom SQL user database like this?
Maybe, but I never heard about it. You could try to wait for the NodeBB core developers. They, probably, know something, that may helps you.
My website is developed using node.js as an express app. Could someone give me any pointers on how I could use NodeBB's user signup / auth system / user session in my express app too?
Is it possible to hook into the NodeBB app so it and my website app run in the same node process and share user session that way? Or would it be better to separate the node processes and communicate user info another way?