how to keep dev forum private from search engines?
-
How can we keep search engines away from our development forum?
Since we always use the up-to-date database of our prod forum, the content is the same and therefore it shows up in the google/duckduckgo searches...
Is it enough to take spider privileges away? What else can we do?
-
Robots.txt
-
@crazycells From recollection, there is a private forum plugin that will force guests to a login screen for all requests, which would also achieve the desired result being nothing to crawl apart from a login page.
-
C crazycells has marked this topic as solved on
-
@PitaJ said in how to keep dev forum private from search engines?:
Robots.txt
I would not rely on such, as many (most?) spiders/bots fail to abide - googlebot being one of, if not the most prominent. Then there's an entire army of various "business intelligence" bots that insist on breaking the rules, cuz hey, they're "exceptional".
fail2ban could be somewhat useful in this regard (sordid history notwithstanding - something is better than nothing), as would be an actual WAF. Both of which require a bit of technical chops to deploy and maintain and hence not for everybody.
I think maybe hosted nodebb uses Varnish (non free)? That would be even slicker yet for those w/the requisite resources.
Have fun!
-
@gotwf this plugin looks like doing all of the job by itself?
GitHub - LM1LC3N7/nodebb-plugin-private-forum: A NodeBB plugin to lockdown the forum to non-registered users.
A NodeBB plugin to lockdown the forum to non-registered users. - LM1LC3N7/nodebb-plugin-private-forum
GitHub (github.com)
-
@crazycells That's exactly the plugin you want - I use this on my dev build
-
@omega said in how to keep dev forum private from search engines?:
What about using user acc privileges to block guests & spiders?
that should also work, but it is easier to activate one plugin to achieve this
-
Hmmm my thoughts are go with what is native and supported that works.
As a general caveat - What happens when the plugin breaks and/or is becomes no longer maintained?
It can happen.
-
Removing view privileges from spiders will get rid of most, but not all crawlers. As @gotwf says, crawlers are not the most law-abiding scripts out there.
Simply removing all guest privileges should do the trick, but if you're regularly copying prod to your dev forum, then the more surefire way to keep EVERYONE away would be to restrict access to a single IP or range (e.g. a VPN) you control.
That might be overkill, though.
