bot to crawl private sections

[JamesC70]

Originally Posted by njoy

there is a site that needs to crawl your site to check links.... However google can see it while testing and the site that needs to crawl sees login page. Anyway to fix this? Once I get it to work is there also a way to only give permission to this site? and not all search engines?

Is Google the site you want to include, or the one to exclude?

Create a robots.txt file, and place it in your sever's web root (wherever www.yoursite.com/index.html is).

In your robots.txt, place the following code:

User-agent: *
Disallow: /

User-agent: Mediapartners-Google
Allow: /

This allowes the AdSense crawler in, but blocks the Googlebot. Replace the "Mediapartners-Google" agent with the agent identifier of whatever other bot you want to crawl your site.

Source: http://www.google.com/support/webmas...y?answer=40360