Have a look at class_core.php. Search for 'enablespiders'. Spiders are decoded in there.
But any use of the built in vB spider system is only going to be as good as your spiders_vbulletin.xml file.
And there are so many spiders out there you'll never get them all. You're better off completely hiding things from unregistered users. Otherwise one spider will feed off of another spiders scan and the content will be picked up anyway.
|