how to keep dev forum private from search engines?

[PitaJ]

Robots.txt

[phenomlab]

@crazycells From recollection, there is a private forum plugin that will force guests to a login screen for all requests, which would also achieve the desired result being nothing to crawl apart from a login page.

[gotwf]

@PitaJ said in how to keep dev forum private from search engines?:

Robots.txt

I would not rely on such, as many (most?) spiders/bots fail to abide - googlebot being one of, if not the most prominent. Then there's an entire army of various "business intelligence" bots that insist on breaking the rules, cuz hey, they're "exceptional".

fail2ban could be somewhat useful in this regard (sordid history notwithstanding - something is better than nothing), as would be an actual WAF. Both of which require a bit of technical chops to deploy and maintain and hence not for everybody.

I think maybe hosted nodebb uses Varnish (non free)? That would be even slicker yet for those w/the requisite resources.

Have fun!

[crazycells]

@gotwf this plugin looks like doing all of the job by itself?

https://github.com/LM1LC3N7/nodebb-plugin-private-forum

[phenomlab]

@crazycells That's exactly the plugin you want - I use this on my dev build

[omega]

What about using user acc privileges to block guests & spiders?

[crazycells]

@omega said in how to keep dev forum private from search engines?:

What about using user acc privileges to block guests & spiders?

that should also work, but it is easier to activate one plugin to achieve this

[omega]

@crazycells

Hmmm my thoughts are go with what is native and supported that works.

As a general caveat - What happens when the plugin breaks and/or is becomes no longer maintained?

It can happen.

[crazycells]

@omega in that case, I will beg here for help

[julian]

Removing view privileges from spiders will get rid of most, but not all crawlers. As @gotwf says, crawlers are not the most law-abiding scripts out there.

Simply removing all guest privileges should do the trick, but if you're regularly copying prod to your dev forum, then the more surefire way to keep EVERYONE away would be to restrict access to a single IP or range (e.g. a VPN) you control.

That might be overkill, though.