As mentioned prior, you can use the privileges to restrict access to the topic.
but I'm not entirely sure how this works.
If a spider is detected (not a 100% guarantee, by the way), then they are either allowed or denied access to the topic based on how your privileges are set up.
For example, if you grant spiders "access-topic" privilege, but do not grant "registered-users" that privilege, then web crawlers will be able to access the topic and crawl it, but regular users will not. The groups are mutually exclusive.
On one hand, I'm thinking that allowing spiders to access all content could potentially boost my site's search engine rankings. On the other hand, I'd like to keep one or two paragraphs hidden from spiders, so that only verified users can view them.
That's harder, you'd want to use a custom plugin to restrict the post content if a spider is detected. Doable, just not available right here right now.
There are plugin hooks that are fired when a post is rendered, and you can probably check caller.uid or req.isSpider (depending on what is passed in) to see whether it is a spider or not. A uid of -1 is a spider as determined by the library we use.
