Blocking spiders

[TitanJeff]

I'm not sure why but my forum had a ton of spiders on it at any given time. I had over 3,000 guest on last week and know the majority were spiders. Is there anything I can do to limit them to certain hours?

[SEOvB]

you can't really pick the time, but chances are good most were the Yahoo spiders. You can place a directive in your robots.txt with a crawl delay for them that will slow them down.

[TitanJeff]

How? I don't believe I have a robots.txt file. Where would that be?

[SEOvB]

its in your website root ex:

http://seotips4.com/robots.txt
http://vbulletin.org/robots.txt

Its never in the forum root. Notice the


User-agent: Slurp
Crawl-delay: 5


Is the line you'll want for your robots.txt file

[TitanJeff]

Many thanks. I'll give it a go.