Yahoo Slurp Invasion!

Originally Posted by no mods

Why are you complaining? the more bots the better.

I once had 2000+ slurps on my server. When you run a big site, and you have 2000+ bots sucking up resources and queries it's certainly NOT a good thing.

Originally Posted by Paul M

Slurpies always visit in packs, one of the forums I admin gets over 2000 a day.

Today for me:

Visitors (2050), Yahoo! Slurp Spiders (1967), Google Spiders (6), MSNBot Spiders (12), AltaVista Spiders (1), AskJeeves Spiders (1), Google AdSense Spiders (1)

2000 slurps over the course of a day, sure. 2000 slurps at once, ack!

[ShawnV]

Anyone running an updated spiders_vbulletin.xml file they would care to share?

_V

[MRGTB]

Mmmmm, I seem to be getting MSN Bots acting like Slurps, in fact Yahoo been a good little boy on my site.

SO FAR! :erm:

[ThunderVamp9]

Yahoo Slurp Spiders have also been running amok on my site, eating up my bandwidth like it was the juices from an insect victim in its web.

Robots/Spiders visitors 	 
Robots	                   Hits	          Bandwidth	           Last visit
Yahoo Slurp	      11763+1054	145.17 MB	12 Jun 2007 - 17:53
Googlebot	         5346+16	      49.91 MB	     12 Jun 2007 - 17:53
MSNBot	                  205+125	        7.90 MB	       12 Jun 2007 - 12:03
Voila	                  83+85	              1.32 MB	     12 Jun 2007 - 15:40
Ask                       41+18	             676.06 KB	   12 Jun 2007 - 17:48
MSNBot-media	         1+18	               10.95 KB	     11 Jun 2007 - 13:14
Alexa (IA Archiver)	   1+1	                 23.72 KB	08 Jun 2007 - 21:11
Lycos	                    1+1	              23.03 KB	     10 Jun 2007 - 07:33
Scooter                     0+1	             609 Bytes	    09 Jun 2007 - 04:53

[inteller2k7]

Where could i get me one of this spiders is that a hack or is that something u have to do to the forums could someone explain i just got ma forums up
aioforums.com

[Zachariah]

Ya they poke their head out a bit.

Googlebot	91838+48	374.35 MB	14 Jun 2007 - 23:51
Inktomi Slurp	25600+2976	210.15 MB	14 Jun 2007 - 23:57
Google AdSense	3784+9	27.17 MB	14 Jun 2007 - 23:20
MSNBot	1987+406	63.68 MB	14 Jun 2007 - 23:46

[Digbijoy]

Originally Posted by ThunderVamp9

Yahoo Slurp Spiders have also been running amok on my site, eating up my bandwidth like it was the juices from an insect victim in its web.

Yeah.

And apart from eating the b/w,
just think of those innumerable queries that are made while indexing those pages.
oo....

But still, we shouldn't mind those spiders/bots/slurps.
Tell me who doesn't want his pages getting indexed by top ranked search engines. ?

[pipin]

Visitors (5.163), Yahoo! Slurp Spiders (2.696), Google Spiders (4), AskJeeves Spiders (1), MSNBot Spiders (12)

well, they dont impact our speed ( me pats the dual 290 opteron combo) but the only thing i care, is that they blow up the user online statistics.

is there a way to exclude the spiders from that?

[loonytune15]

i just wish i could get my google spiders on my site..

[Marcimi]

I'm just running a small board used by a couple of ppls (usually around 20 ppl's) but that Yahoo-Thing tries to explode my b/w and my statistics.

I tried to exclude them with an updated robot.txt but nothing happens :/

Is there another way to exclude them? We do not have any public informations that is good enough to be listed on yahoo (or another search engine).