bot to crawl private sections

[njoy]

Ok before you say why do you want to do this let me explain. I do not want to do this for search engines, there is a site that needs to crawl your site to check links. Now I have installed kx permissions mod but it does not seem to work for me. Gave all permissions in options for the mod and edited the vbulletin spiders file to include the bot I need to crawl the private sections of my site. However google can see it while testing and the site that needs to crawl sees login page. Anyway to fix this? Once I get it to work is there also a way to only give permission to this site? and not all search engines?

[njoy]

no one?

[JamesC70]

Originally Posted by njoy

there is a site that needs to crawl your site to check links.... However google can see it while testing and the site that needs to crawl sees login page. Anyway to fix this? Once I get it to work is there also a way to only give permission to this site? and not all search engines?

Is Google the site you want to include, or the one to exclude?

Create a robots.txt file, and place it in your sever's web root (wherever www.yoursite.com/index.html is).

In your robots.txt, place the following code:

User-agent: *
Disallow: /

User-agent: Mediapartners-Google
Allow: /

This allowes the AdSense crawler in, but blocks the Googlebot. Replace the "Mediapartners-Google" agent with the agent identifier of whatever other bot you want to crawl your site.

Source: http://www.google.com/support/webmas...y?answer=40360