vb.org Archive - Yahoo Slurp Invasion!

Page 4 of 5

« First

Show 40 post(s) from this thread on one page

vb.org Archive (https://vborg.vbsupport.ru/index.php)

- Community Lounge (https://vborg.vbsupport.ru/forumdisplay.php?f=13)

- - Yahoo Slurp Invasion! (https://vborg.vbsupport.ru/showthread.php?t=149331)

ShawnV

07-11-2007 12:51 AM

I have had over 2,200 yahoo spiders assault my site in a day, and was averaging 1,700, I finally got sick of it and set their crawl delay to 1000, the most I see now is 200-300 a day. I don't see a need for more then that to index my content since the Google spider does it with just 1.

_v

Alfa1

07-11-2007 03:38 PM

At least this hack sets Who's online straight. It enables you to get a clear view on the amount of people you have online.

mfyvie

07-15-2007 12:22 PM

I have been really irritated by the number of bots crawling my site. My irritation was from two sources:

1. Useful bots are welcome - e.g. google, yahoo, etc, but most of the rest just steal content and slow your site down. I banned about 200 known bot types by blocking them at my web server. My server load when down immediately...
2. Statistics - as you all know it looks stupid if you are showing 1000 guests that aren't really guests. I wrote this hack to deal with the problem. It's a little different to the one linked above. You should check both hacks out and decide which suits you better.

ragtek

07-15-2007 12:54 PM

Quote:

Originally Posted by Zachariah (Post 1269392)

Ya they poke their head out a bit.

Code:

Googlebot        91838+48        374.35 MB        14 Jun 2007 - 23:51

Inktomi Slurp        25600+2976        210.15 MB        14 Jun 2007 - 23:57

Google AdSense        3784+9        27.17 MB        14 Jun 2007 - 23:20

MSNBot        1987+406        63.68 MB        14 Jun 2007 - 23:46

where do you see this information?

FreshFroot

07-16-2007 05:16 AM

yeah I once had 800 in 1 day... slowed the server down by a lot...

G0F0RBR0KE

07-16-2007 05:46 AM

I had one time over 400 on my old domain. Get the IP and block them from the site.

theFAILURE

07-16-2007 03:47 PM

I was thinking that the Yahoo Slurp! Spider had calmed down a bit, but I was very wrong lol.

https://vborg.vbsupport.ru/external/2007/07/29.jpg

iogames

08-05-2007 03:48 AM

Does anyone has experienced a shutdown on server due to this factor?

StrifeX

08-09-2007 08:14 AM

Lol, you just need to have good forum posts which are popular and they will just popup once or twice a day :p And they aren't that bad, just get rid of the active users link and your set with 100's of users which look like regular guests, which IMO makes people think the site is more popular than it actually is and makes them want to stay :D

Alfa1

05-30-2008 01:31 PM

This is getting a bit insane now. I am regularly getting over 2100 yahoo slurp spiders at the same time. My server can take it, but I am worried how much this is costing me. I am aware that I can delay the indexing by slurp spiders, but I do not see reason to do this. If there is anyone who can explain this,then that would be very helpful.

At the same time, I do not want to suppress Yahoo! from getting my site high up in their search index. Does anybody know what to do with this?

All times are GMT. The time now is 08:23 AM.

Page 4 of 5

« First

Show 40 post(s) from this thread on one page

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.01953 seconds Memory Usage 1,734KB Queries Executed 10 (?)
More Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (1)bbcode_code_printable (1)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)pagenav (1)pagenav_curpage (3)pagenav_pagelink (1)post_thanks_navbar_search (1)printthread (10)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start pagenav_page pagenav_complete bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages: