vb.org Archive - Yahoo Slurp Invasion!

Page 1 of 2

Show 40 post(s) from this thread on one page

vb.org Archive (https://vborg.vbsupport.ru/index.php)

- Community Lounge (https://vborg.vbsupport.ru/forumdisplay.php?f=13)

- - Yahoo Slurp Invasion! (https://vborg.vbsupport.ru/showthread.php?t=149331)

ShawnV

06-09-2007 11:33 PM

Yahoo Slurp Invasion!

Yahoo! Slurp Spiders (1007), MSNBot Spiders (11), Google Spiders (1)

And thats with a robots.txt of:

User-agent: Slurp
Crawl-delay: 60

Yahoo needs to be taken out behind the wood shed and beaten to a pulp for this BS.

_V

nexialys

06-10-2007 12:29 AM

yahoo slurps often because you have a popular topic... no matter the robots.txt ...

and i suppose it will come with a non-robot taff soon, because they are trying to overcome google...

HMBeaty

06-10-2007 12:37 AM

Wow, send some of those to me, LOL

ShawnV

06-10-2007 01:25 AM

Quote:

Originally Posted by Redlinemotorsports (Post 1265178)

Wow, send some of those to me, LOL

Up to since last posting: Yahoo! Slurp Spiders (1067), MSNBot Spiders (11), Google Spiders (1)

You can have them all, please take them. ;)

_V

no mods

06-10-2007 02:21 AM

Why are you complaining? the more bots the better.:)

theFAILURE

06-10-2007 02:54 AM

I have about 100 of them online now; for the day would be over 2000 for sure.

I don't mind :) they do not seem to take up much resources.

Are you feeling an impact?

Ba$im

06-10-2007 03:18 AM

its there r somthing do to get like u friends?

Paul M

06-10-2007 09:03 AM

Slurpies always visit in packs, one of the forums I admin gets over 2000 a day.

Quote:

Visitors (5,540), Yahoo! Slurp Spiders (2,304), Google Spiders (11), AltaVista Spiders (1), AskJeeves Spiders (2), MSNBot Spiders (15), Google AdSense Spiders (1)

Frozen Tears

06-10-2007 10:27 AM

Thats alot of spiders =o

ShawnV

06-10-2007 10:56 AM

Quote:

Originally Posted by Paul M (Post 1265395)

Slurpies always visit in packs, one of the forums I admin gets over 2000 a day.

Seems so, I broke 2,873 Slurps last night, guess it doesn't really matter they cant be using up that many resources.

It just smells of desperation for Yahoo to be flooding the web with that many spiders.

_V

Guest210212002

06-10-2007 02:51 PM

Quote:

Originally Posted by no mods (Post 1265258)

Why are you complaining? the more bots the better.:)

I once had 2000+ slurps on my server. When you run a big site, and you have 2000+ bots sucking up resources and queries it's certainly NOT a good thing.

Quote:

Originally Posted by Paul M (Post 1265395)

Slurpies always visit in packs, one of the forums I admin gets over 2000 a day.

Today for me:

Quote:

Visitors (2050), Yahoo! Slurp Spiders (1967), Google Spiders (6), MSNBot Spiders (12), AltaVista Spiders (1), AskJeeves Spiders (1), Google AdSense Spiders (1)

2000 slurps over the course of a day, sure. 2000 slurps at once, ack!

ShawnV

06-12-2007 10:25 PM

Anyone running an updated spiders_vbulletin.xml file they would care to share?

_V

MRGTB

06-13-2007 01:53 AM

Mmmmm, I seem to be getting MSN Bots acting like Slurps, in fact Yahoo been a good little boy on my site.

SO FAR! :erm:

ThunderVamp9

06-13-2007 11:57 AM

Yahoo Slurp Spiders have also been running amok on my site, eating up my bandwidth like it was the juices from an insect victim in its web.

Code:

Robots/Spiders visitors          

Robots                           Hits                  Bandwidth                   Last visit

Yahoo Slurp              11763+1054        145.17 MB        12 Jun 2007 - 17:53

Googlebot                 5346+16              49.91 MB             12 Jun 2007 - 17:53

MSNBot                          205+125                7.90 MB               12 Jun 2007 - 12:03

Voila                          83+85                      1.32 MB             12 Jun 2007 - 15:40

Ask                       41+18                     676.06 KB           12 Jun 2007 - 17:48

MSNBot-media                 1+18                       10.95 KB             11 Jun 2007 - 13:14

Alexa (IA Archiver)           1+1                         23.72 KB        08 Jun 2007 - 21:11

Lycos                            1+1                      23.03 KB             10 Jun 2007 - 07:33

Scooter                     0+1                     609 Bytes            09 Jun 2007 - 04:53

inteller2k7

06-15-2007 08:48 PM

Where could i get me one of this spiders is that a hack or is that something u have to do to the forums could someone explain i just got ma forums up
aioforums.com

Zachariah

06-15-2007 11:00 PM

Ya they poke their head out a bit.

Code:

Googlebot        91838+48        374.35 MB        14 Jun 2007 - 23:51

Inktomi Slurp        25600+2976        210.15 MB        14 Jun 2007 - 23:57

Google AdSense        3784+9        27.17 MB        14 Jun 2007 - 23:20

MSNBot        1987+406        63.68 MB        14 Jun 2007 - 23:46

Digbijoy

06-25-2007 08:05 AM

Quote:

Originally Posted by ThunderVamp9 (Post 1267630)

Yahoo Slurp Spiders have also been running amok on my site, eating up my bandwidth like it was the juices from an insect victim in its web.

Yeah.

And apart from eating the b/w,
just think of those innumerable queries that are made while indexing those pages. :eek:
oo....

But still, we shouldn't mind those spiders/bots/slurps.
Tell me who doesn't want his pages getting indexed by top ranked search engines. ?

pipin

06-26-2007 07:43 AM

Quote:

Visitors (5.163), Yahoo! Slurp Spiders (2.696), Google Spiders (4), AskJeeves Spiders (1), MSNBot Spiders (12)

well, they dont impact our speed ( me pats the dual 290 opteron combo) but the only thing i care, is that they blow up the user online statistics.

is there a way to exclude the spiders from that?

loonytune15

06-26-2007 08:12 AM

i just wish i could get my google spiders on my site..

Marcimi

06-30-2007 01:40 PM

I'm just running a small board used by a couple of ppls (usually around 20 ppl's) but that Yahoo-Thing tries to explode my b/w and my statistics.

I tried to exclude them with an updated robot.txt but nothing happens :/

Is there another way to exclude them? We do not have any public informations that is good enough to be listed on yahoo (or another search engine).

Guest210212002

06-30-2007 01:43 PM

Quote:

Originally Posted by inteller2k7 (Post 1269321)

Where could i get me one of this spiders is that a hack or is that something u have to do to the forums could someone explain i just got ma forums up
aioforums.com

Spiders are content indexers from search engines. They're automated "bots", so to speak, that crawl your site and pick up relevant content, keywords.

Submit your site to search engines (google "search engine submission") - they'll come in time.

Marcimi

06-30-2007 01:58 PM

Is there a way to reset the "User Online" Statistics? Because a few thousand user's being online are not exactly the truth (our record is around 100).

Dream

07-01-2007 06:45 AM

Yes there is

https://vborg.vbsupport.ru/showthread.php?t=130961

cheat-master30

07-01-2007 01:21 PM

I had this earlier:

http://www.dsultimate.net/Avatar/43bots.PNG

Then another 5 of them came. Now, considering my board at max gets like 15 members online at once, those Yahoo bots have tripled the user online count from what it's meant to be.

Dream

07-01-2007 02:02 PM

I just had 77 Slurp spiders on a forum with 24 topics.

cheat-master30

07-01-2007 04:18 PM

Quote:

Originally Posted by Dream (Post 1280637)

I just had 77 Slurp spiders on a forum with 24 topics.

What content could they possibly index in 24 topics?

And for anyone else, why are so many Yahoo Spiders on forums? And they don't seem to obey bot restrictions.

But funnier was this, that I had a NEGATIVE amount of guests this morning (a few members, a few bots, -1 guests)

theFAILURE

07-01-2007 06:47 PM

https://vborg.vbsupport.ru/external/2007/07/49.jpg

Heh, yesterday there was over 200 of the little bastards, today is not so bad though :)

I generally do not mind, they do not impact performance and I own my own server.

I'm #1 on some in some of my desired search queries as well.

I would assume that if someone has a bandwidth limitation it would suck though.

Alfa1

07-01-2007 07:01 PM

Just had 1.600 of them. Cheat master: which add-on do you use to display the slurp spiders separately?

cheat-master30

07-02-2007 04:51 AM

Quote:

Originally Posted by Alfa1 (Post 1280797)

Just had 1.600 of them. Cheat master: which add-on do you use to display the slurp spiders separately?

https://vborg.vbsupport.ru/showthread.php?t=145328

That modification.

cheat-master30

07-10-2007 10:26 AM

And sorry to bump this, but I've had another Yahoo Slurp attack on my forum (see picture):

http://www.dsultimate.net/Avatar/yahoogonemad.PNG

Out of those 28 bots, 27 are from Yahoo, and 1 is from Google. Nice comparison of how Yahoo slurp invades compared to other search engine spiders...

ShawnV

07-11-2007 12:51 AM

I have had over 2,200 yahoo spiders assault my site in a day, and was averaging 1,700, I finally got sick of it and set their crawl delay to 1000, the most I see now is 200-300 a day. I don't see a need for more then that to index my content since the Google spider does it with just 1.

_v

Alfa1

07-11-2007 03:38 PM

At least this hack sets Who's online straight. It enables you to get a clear view on the amount of people you have online.

mfyvie

07-15-2007 12:22 PM

I have been really irritated by the number of bots crawling my site. My irritation was from two sources:

1. Useful bots are welcome - e.g. google, yahoo, etc, but most of the rest just steal content and slow your site down. I banned about 200 known bot types by blocking them at my web server. My server load when down immediately...
2. Statistics - as you all know it looks stupid if you are showing 1000 guests that aren't really guests. I wrote this hack to deal with the problem. It's a little different to the one linked above. You should check both hacks out and decide which suits you better.

ragtek

07-15-2007 12:54 PM

Quote:

Originally Posted by Zachariah (Post 1269392)

Ya they poke their head out a bit.

Code:

Googlebot        91838+48        374.35 MB        14 Jun 2007 - 23:51

Inktomi Slurp        25600+2976        210.15 MB        14 Jun 2007 - 23:57

Google AdSense        3784+9        27.17 MB        14 Jun 2007 - 23:20

MSNBot        1987+406        63.68 MB        14 Jun 2007 - 23:46

where do you see this information?

FreshFroot

07-16-2007 05:16 AM

yeah I once had 800 in 1 day... slowed the server down by a lot...

G0F0RBR0KE

07-16-2007 05:46 AM

I had one time over 400 on my old domain. Get the IP and block them from the site.

theFAILURE

07-16-2007 03:47 PM

I was thinking that the Yahoo Slurp! Spider had calmed down a bit, but I was very wrong lol.

https://vborg.vbsupport.ru/external/2007/07/29.jpg

iogames

08-05-2007 03:48 AM

Does anyone has experienced a shutdown on server due to this factor?

StrifeX

08-09-2007 08:14 AM

Lol, you just need to have good forum posts which are popular and they will just popup once or twice a day :p And they aren't that bad, just get rid of the active users link and your set with 100's of users which look like regular guests, which IMO makes people think the site is more popular than it actually is and makes them want to stay :D

Alfa1

05-30-2008 01:31 PM

This is getting a bit insane now. I am regularly getting over 2100 yahoo slurp spiders at the same time. My server can take it, but I am worried how much this is costing me. I am aware that I can delay the indexing by slurp spiders, but I do not see reason to do this. If there is anyone who can explain this,then that would be very helpful.

At the same time, I do not want to suppress Yahoo! from getting my site high up in their search index. Does anybody know what to do with this?

All times are GMT. The time now is 01:40 AM.

Page 1 of 2

Show 40 post(s) from this thread on one page

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.01926 seconds Memory Usage 1,826KB Queries Executed 10 (?)
More Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (3)bbcode_code_printable (12)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)pagenav (1)pagenav_curpage (1)pagenav_pagelink (1)post_thanks_navbar_search (1)printthread (40)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start pagenav_page pagenav_complete bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages: