How do I limit the # of google and yahoo crawler bots?
 

I'm suddenly getting 10-15 google / yahoo bots at a time. How do I limit the number of bot connections?

Normally you only see one or two Google spiders at a time. However with Yahoo you can see dozens or even hundreds of spiders. Here is an example of a robots.txt which only allows Google and Bing to index your forum.

User-agent: Googlebot
Disallow:

User-agent: MSNBot
Disallow:

User-agent: *
Disallow: /

in reverse of this "issue/Problem" how do you make your site more visible to these bots/spiders to improve searchability?

Thanks
Ben

glenn, go to google.com/webmasters and then after you've configured your site for it, click your site and then

*Site configuration
*Settings
*Crawl Rate, you can set the crawl rate to either be faster or slower

Thanks guys... I'm still bringing my site up to speed and am not ready for the swarm of crawlers. Nice to know that my site is hitting the top of the search list on the keywords I use :D

I wouldn't completely disallow a spider. After all, you want them to index things. I used this instead:
Slurp was the only one I saw with an inordinate number of connections (in the hundreds).

Is there any way to disallow google from indexing one specific forum using robots.txt, rather than just enable/disabling it??

Thx
Paul

I've always understood that the bots all browse the forum with Guest permissions (I hope!), so you can just disallow Guests from seeing that forum.