Will the Slurp ever calm down?
 

Does anyone know if Yahoo is going to calm down with the attack of the Yahoo Slurp Spider?

Tonight I had quite a few of them:

https://vborg.vbsupport.ru/external/2007/07/6.jpg

On boards with a shared hosting packages, it must really suck.

If you have limited bandwidth, it also must kill you.

Tonight they almost beat my "most users ever online" stat!

you can throttle the slurp in your robots file

it's not a big amoung of bandwidth actually, because each slurp is a page hit... yahoo is not using the same technique as google... google is more to fear actually because if you have more than one google connection, you have some bots hitting your board at the same time...

I read on several occasions that they are ignoring the robots.txt file.

True?

Wow. That's a whole lot of spiders you have online, tF. I'm stll waiting for the day that many decide to hit my boards. ;)

I have realized that there have been a WHOLE LOT MORE spiders since I turned on the "forum archive".

This leads me to believe if you are having a problem with the spiders and have the forum archive turned on, you can turn it off in your admincp to release some pressure if there is any.

forum archive is actually the best way for search engine to access your site... this is the quickest and simpliest way, but also better for your server, and the archive is 10 times less server-intense...

With that in mind, is there a way to keep them off regular threads, and keep them crawling through the Archives only?

i suppose there is a hack that control that... did not search.

Essentially if its the same EXACT result, I can not see why anyone with Archiving turned on would not opt for it :)