And use Two-Factor Authentication (2FA) .
How can we keep search engines away from our development forum?
Since we always use the up-to-date database of our prod forum, the content is the same and therefore it shows up in the google/duckduckgo searches...
Is it enough to take spider privileges away? What else can we do?
I would not rely on such, as many (most?) spiders/bots fail to abide - googlebot being one of, if not the most prominent. Then there's an entire army of various "business intelligence" bots that insist on breaking the rules, cuz hey, they're "exceptional".
fail2ban could be somewhat useful in this regard (sordid history notwithstanding - something is better than nothing), as would be an actual WAF. Both of which require a bit of technical chops to deploy and maintain and hence not for everybody.
I think maybe hosted nodebb uses Varnish (non free)? That would be even slicker yet for those w/the requisite resources.
Removing view privileges from spiders will get rid of most, but not all crawlers. As @gotwf says, crawlers are not the most law-abiding scripts out there.
Simply removing all guest privileges should do the trick, but if you're regularly copying prod to your dev forum, then the more surefire way to keep EVERYONE away would be to restrict access to a single IP or range (e.g. a VPN) you control.
That might be overkill, though.