| memobug |
11-28-2005 08:09 PM |
robots.txt - can't stop slurp!
I am running my vB forum in a subdomain. I added robots.txt to both my root and subdomain folder, but I can't seem to stop slurp (inktomi) from trying to access all the wrong pages.
in my subdomain ( http://forum.mydomain.com/robots.txt) , I didn't know if I needed the preceding / so I have
Quote:
User-agent: *
Disallow: /attachment.php
Disallow: /newattachment.php
Disallow: /avatar.php
Disallow: /editpost.php
Disallow: /login.php
Disallow: /member.php
Disallow: /member2.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newreply.php
Disallow: /newthread.php
etc.
|
Do I have the path wrong Should I take out the leading slash / ? The spider seems to be ignoring my disallows and its trying to edit posts, visit the print views and all the other banned stuff. The path to my forum is like http://forum.mydomain.com/index.php
in my root ( http://www.mydomain.com/robots.txt) Just in case, I also I have
Quote:
User-agent: *
Disallow: /forums/attachment.php
Disallow: /forums/newattachment.php
Disallow: /forums/avatar.php
Disallow: /forums/editpost.php
Disallow: /forums/login.php
Disallow: /forums/member.php
Disallow: /forums/member2.php
Disallow: /forums/misc.php
etc.
|
|
More Information