Yahoo Slurp Spiders

[MetalMilitia]

http://www.armageddononline.org
http://forums.armageddononline.org

I run a fairly "mid level" site.... ~3000 unique hits / day....

What is the logic behind Yahoo having 500 or so spiders on the forums at one time? Google seems to get the job done better with 5-10 / day....

I know there are other threads, but if someone could lay it all out here I would appreciate it much.



-Matt

[snakes1100]

You would have to ask yahoo why they cant figure out how to send an appropriate amount of bots to a new site they are seo'ing

You can limit the bots by using a robots.txt file and setting limits for the yahoo bots.

[MetalMilitia]

I know you can change the times / limits.... but I mean good lord

500+ on easter sunday morning? lols.

I'm sure it's a healthy testament to the server, but why the hell do then need 200x more crawlers than google... from which gives me 90% of the traffic anyways?

It's like yahoo hits a dead end somewhere (per say a closed thread or error) - and then proceeds to call in 5 more spiders to check out why. Is it not somewhat ridiculous?

How bad do you guys with bigger sites get hit?

-MM-

[snakes1100]

It's really hard to say why yahoo sends in 5 platoons of troops to crawl a site, it's still a question they would have to answer.

I've encountered that on a few sites i admin for, amending the robots file will stop them, but if i recall its a 30 day waiting period for yahoo to check the file again, an easy and fast way to get rid of them is to ban all but one of yahoo bot ips, i think they use about 30 ips to.

[MetalMilitia]

Yup.

As much as it bothers me watching them do 200x the work google does, I don't exactly want to trash a search engine hit of any kind.

.... but there in lies the question that no one seems to know the answer to. Why DOES yahoo do that? As stated, and hits to the boards of keywords come from primarily (75%+) google.... same with the main site and articles / news.

If I only get ~3000 unique hits / day for everywhere on the site, how bad do bigger sites get hit?

[snakes1100]

I doubt your going to get a real answer as to why yahoo bots do that, you could search google for an answer though

Most big sites are going to implement a robots.txt file and not worry about it any further once yahoo reads the file, the bots will comply.

[MetalMilitia]

I still leave the question open for those that want to answer though...

What is the most ridiculous amount of bots / spiders you have seen on your boards? Include the size of the boards too please

-MM

To stop Slurp's, er, slurping, toss this in your robots.txt:

User-agent: Slurp
Crawl-delay: 60

My site today, and it's only 9:40AM.

Guest Visits Today: 4,667
Visitors (3813), Yahoo! Slurp Spiders (813), Google AdSense Spiders (8), Google Spiders (23), AskJeeves Spiders (1), MSNBot Spiders (9)