Yahoo! Slurp Spider, Sucking Us Dry

[Tiberonmagnus]

I'm trying to get some slurps, come to pappa!! Added my forums to a bunch of search engines.. just wondering how long it will take.

[Nomadite]

Originally Posted by Tiberonmagnus

I'm trying to get some slurps, come to pappa!! Added my forums to a bunch of search engines.. just wondering how long it will take.

On one site that I administer, we get an average of 50-90 Slurps at any given time along with the usual fanfare of other search bots. But it's all good because each is spidering different section.

On my own forum (a different forum), which is about 5 months old, I've noticed that each time I add new content (cellphone wallpapers) I get hit with anywhere from 2 - 5 of them within a few mintutes of the new upload, almost without fail and they go directly to the section of the forum where I added the new content.

If I get some spare Sluprs I'll send 'em your way .

[da420]

Send some of those guys my way

[Artificial_Alex]

Gaminggutter.com has atleast 40-50 spiders on at once. :P

[Andem]

You think 15 spiders online is bad? Slurp has had about 500 spiders on my site since yesterday; they're still there.. around double the normal amount!

User-agent: Slurp
Crawl-delay: 60

Has solved all of my woes. I'm not sure what's been up in the last week, but with any other CD I've had hundreds of them online. I've no idea WHY that works, or if it will work for someone else, but it's working for me. (knock on wood)

Edit: It does still ignore my disallows though. :\

On the upside, I had 13,947 indexed pages in Yahoo two weeks ago, and currently I have 48,000, albeit after a LOT of SEO work, but it does seem like they're updating their content a bit more rigorously.

[Sownman]

I agree that spiders are a problem. I am having trouble with my host for using too much resource because of too many persistant DB connections. I always have lots of spiders.
I don't want to dissallow them in robots.txt as they are good. But, I must cut back on DB connections and I don't want to have 30 Yahoo spiders all at once while members can't get on.

IS THERE ANYTHING that will limit Slurps to ten at a time anyway. ??

Thanks

Steve

[Lynne]

Originally Posted by Sownman

IS THERE ANYTHING that will limit Slurps to ten at a time anyway. ??

Thanks

Steve

Yes, put in the crawl delay as suggested above. If you have your cookie timeout set to 900 seconds and you only want 10 of those spiders on at a time, then set the crawl delay to 900/10=90.

[Sownman]

Originally Posted by Lynne

Yes, put in the crawl delay as suggested above. If you have your cookie timeout set to 900 seconds and you only want 10 of those spiders on at a time, then set the crawl delay to 900/10=90.

I'll try that, thanks. Higher number should equal fewer spiders ?

Steve

[ZomgStuff]

I once had about 800 yaho0 slurpp spiders at one time, I was panicking, thought it was a Ddos.