This is a no brainer!
I am running 2 very small Wordpress Blogs on Ubuntu 14.04 LEMP with 1GB RAM on digitalocean
I have little knowledge on this. Please bear with me.
-Is there any guide on how to install nodeBB under this configuration? nodeBB does not have to link with wordpress. It will run under "/forum" or "forum.aaaa.com"
JuSt install normally and put on forum.domain.com. Won't interfere with WordPress.
Thank you. Which DB should I use? Redis or MongoDB?
Up to you, Redis is faster but costs more per gb in terms of storage, if you plan on having a lot of posts then mongodb. I had just under 30k posts on a 512mb droplet without issue. So, redis would be fine for you.