Removing view privileges from spiders will get rid of most, but not all crawlers. As @gotwf says, crawlers are not the most law-abiding scripts out there.
Simply removing all guest privileges should do the trick, but if you're regularly copying prod to your dev forum, then the more surefire way to keep EVERYONE away would be to restrict access to a single IP or range (e.g. a VPN) you control.
That might be overkill, though.
