How Can I Slow Down Yahoo Spiders?

[paldo]

this beats anything i have ever seen yahoo spiders have a love affair with my forums most people are happy to have a spider or 3 but i have had as many as 600 yahoo spiders on my board at once .. i have played with a robots.txt and the only way i can slow them down is by disallowing access to my forums which defeats the purpose of having spiders on your forums when they cant read it .. i dont mind them being there but a few would be nice rather then by the 100's

Currently Active Users: 240 (9 members and 231 guests)

http://img131.imageshack.us/img131/4852/spidersgr0.png

sinse i posted this and refreshed my page here is the new count ..




Visitors (27), Google Spiders (1), Yahoo! Slurp Spiders (504)
its insane

[Zachery]

Give them a day or two and they wil wonder off.

[paldo]

thats what i thought this has been a battle for over a month .my site has 200-300 new post a day but this is nuts lol

Currently Active Users: 342 (9 members and 333 guests)

i now see this site has alot of yahoo bots online as well i just dont see why they hafta crawl in masses like that

[SaN-DeeP]

check the robots.txt file for this site -> vbulletin.com/forums/robots.txt
you can setup a crawl delay.