Log in

View Full Version : Will the Slurp ever calm down?

theFAILURE
07-26-2007, 11:57 PM
Does anyone know if Yahoo is going to calm down with the attack of the Yahoo Slurp Spider?

Tonight I had quite a few of them:

https://vborg.vbsupport.ru/external/2007/07/6.jpg

On boards with a shared hosting packages, it must really suck.

If you have limited bandwidth, it also must kill you.

Tonight they almost beat my "most users ever online" stat!
d8tabyte
07-27-2007, 12:12 AM
you can throttle the slurp in your robots file
nexialys
07-27-2007, 12:13 AM
it's not a big amoung of bandwidth actually, because each slurp is a page hit... yahoo is not using the same technique as google... google is more to fear actually because if you have more than one google connection, you have some bots hitting your board at the same time...
theFAILURE
07-27-2007, 01:03 AM
you can throttle the slurp in your robots file

I read on several occasions that they are ignoring the robots.txt file.

True?
projectego
07-27-2007, 08:01 AM
Wow. That's a whole lot of spiders you have online, tF. I'm stll waiting for the day that many decide to hit my boards. ;)
theFAILURE
07-27-2007, 04:57 PM
I have realized that there have been a WHOLE LOT MORE spiders since I turned on the "forum archive".

This leads me to believe if you are having a problem with the spiders and have the forum archive turned on, you can turn it off in your admincp to release some pressure if there is any.
nexialys
07-27-2007, 05:49 PM
forum archive is actually the best way for search engine to access your site... this is the quickest and simpliest way, but also better for your server, and the archive is 10 times less server-intense...
theFAILURE
07-27-2007, 05:59 PM
With that in mind, is there a way to keep them off regular threads, and keep them crawling through the Archives only?
nexialys
07-27-2007, 06:07 PM
i suppose there is a hack that control that... did not search.
theFAILURE
07-27-2007, 06:11 PM
Essentially if its the same EXACT result, I can not see why anyone with Archiving turned on would not opt for it :)
theFAILURE
07-29-2007, 02:42 AM
It looks like the slurp has calmed down!

I noticed it on my site, as we as vb :)

Scratch that, they are back here and on my site in full force again.