Log in

View Full Version : spiders (web crawlers) - need help


sandrodz
12-13-2006, 08:52 PM
Hey, I noticed several crawlers like yahoo and google where indexing pages that don't contain any info like:

Yahoo! Slurp Spider
Viewing 'No Permission' Message /calendar.php?do=add&c=1&day=2007-1-19 Creating Event

is there any mod that would lock away some pages of the forum? any help appreciated!

Paul M
12-13-2006, 09:52 PM
No permission means it's locked away. :)

Create a robots.txt file to limit what they try and view in the first place.

SaN-DeeP
12-15-2006, 12:37 AM
Its strange here as well, often I have noticed Bots (specially google) showing no permission mesage in whois online for our phootopost gallery but when a user clicks on this link this works fine. :)