Seeking Advice on Configuring NodeBB for Exclusive Content Access by Verified Users and Spiders
-
Greetings everyone,
I'm seeking advice on the best way to share some content exclusively with registered and verified users on my website. I understand that I can configure the privileges setting for different user roles, including spiders, but I'm not entirely sure how this works.
On one hand, I'm thinking that allowing spiders to access all content could potentially boost my site's search engine rankings. On the other hand, I'd like to keep one or two paragraphs hidden from spiders, so that only verified users can view them.
My question is: How can I achieve this using NodeBB? Specifically, how can I grant spiders default access to my site's content while also ensuring that certain hidden files can only be accessed by verified users?
Any suggestions or guidance would be greatly appreciated. Thank you in advance.
-
Hi Carol,
Users and robots are handled separately.
Users: the easiest way to show some content only to registered users is by putting it in specific categories, then only granting read permissions for those categories to registered users. People will have to be logged in to access them.
Robots: by default, automated entities like Google web spiders can access anything a non-logged-in user can access. They can't log into your site, and therefore can't see anything in a "registered users only" category.
You can further dictate what a bot should(n't) access in the robots.txt file and/or robots meta tags.
Also, robots see all of a document or nothing. You can't hide parts of a doc from them.
Hope that helps!
-
As mentioned prior, you can use the privileges to restrict access to the topic.
but I'm not entirely sure how this works.
If a spider is detected (not a 100% guarantee, by the way), then they are either allowed or denied access to the topic based on how your privileges are set up.
For example, if you grant spiders "access-topic" privilege, but do not grant "registered-users" that privilege, then web crawlers will be able to access the topic and crawl it, but regular users will not. The groups are mutually exclusive.
On one hand, I'm thinking that allowing spiders to access all content could potentially boost my site's search engine rankings. On the other hand, I'd like to keep one or two paragraphs hidden from spiders, so that only verified users can view them.
That's harder, you'd want to use a custom plugin to restrict the post content if a spider is detected. Doable, just not available right here right now.
There are plugin hooks that are fired when a post is rendered, and you can probably check
caller.uidorreq.isSpider(depending on what is passed in) to see whether it is a spider or not. A uid of-1is a spider as determined by the library we use.
