Go Back   vb.org Archive > Community Central > Community Lounge
Reload this Page Yahoo Slurp Invasion!
FAQ Community Calendar Today's Posts Search

Reply
Page 4 of 5 « First 23 4 5
 
Thread Tools Display Modes
  #31  
Old 07-11-2007, 01:51 AM
ShawnV's Avatar
ShawnV ShawnV is offline
 
Join Date: Mar 2005
Posts: 270
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
I have had over 2,200 yahoo spiders assault my site in a day, and was averaging 1,700, I finally got sick of it and set their crawl delay to 1000, the most I see now is 200-300 a day. I don't see a need for more then that to index my content since the Google spider does it with just 1.

_v
Reply With Quote
  #32  
Old 07-11-2007, 04:38 PM
Alfa1's Avatar
Alfa1 Alfa1 is offline
 
Join Date: Dec 2005
Location: Netherlands
Posts: 3,537
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
At least this hack sets Who's online straight. It enables you to get a clear view on the amount of people you have online.
Reply With Quote
  #33  
Old 07-15-2007, 01:22 PM
mfyvie mfyvie is offline
 
Join Date: Mar 2007
Location: Zurich, Switzerland
Posts: 336
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
I have been really irritated by the number of bots crawling my site. My irritation was from two sources:

1. Useful bots are welcome - e.g. google, yahoo, etc, but most of the rest just steal content and slow your site down. I banned about 200 known bot types by blocking them at my web server. My server load when down immediately...
2. Statistics - as you all know it looks stupid if you are showing 1000 guests that aren't really guests. I wrote this hack to deal with the problem. It's a little different to the one linked above. You should check both hacks out and decide which suits you better.
Reply With Quote
  #34  
Old 07-15-2007, 01:54 PM
ragtek ragtek is offline
 
Join Date: Mar 2006
Location: austria, croatia
Posts: 1,630
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Quote:
Originally Posted by Zachariah View Post
Ya they poke their head out a bit.

Code:
Googlebot	91838+48	374.35 MB	14 Jun 2007 - 23:51
Inktomi Slurp	25600+2976	210.15 MB	14 Jun 2007 - 23:57
Google AdSense	3784+9	27.17 MB	14 Jun 2007 - 23:20
MSNBot	1987+406	63.68 MB	14 Jun 2007 - 23:46
where do you see this information?
Reply With Quote
  #35  
Old 07-16-2007, 06:16 AM
FreshFroot's Avatar
FreshFroot FreshFroot is offline
 
Join Date: Jul 2005
Posts: 770
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
yeah I once had 800 in 1 day... slowed the server down by a lot...
Reply With Quote
  #36  
Old 07-16-2007, 06:46 AM
G0F0RBR0KE G0F0RBR0KE is offline
 
Join Date: Mar 2005
Posts: 987
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
I had one time over 400 on my old domain. Get the IP and block them from the site.
Reply With Quote
  #37  
Old 07-16-2007, 04:47 PM
theFAILURE's Avatar
theFAILURE theFAILURE is offline
 
Join Date: Jan 2005
Posts: 141
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
I was thinking that the Yahoo Slurp! Spider had calmed down a bit, but I was very wrong lol.

Reply With Quote
  #38  
Old 08-05-2007, 04:48 AM
iogames's Avatar
iogames iogames is offline
 
Join Date: Jan 2007
Location: Las Vegas, NV.
Posts: 1,433
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Does anyone has experienced a shutdown on server due to this factor?
Reply With Quote
  #39  
Old 08-09-2007, 09:14 AM
StrifeX StrifeX is offline
 
Join Date: Dec 2006
Location: New York
Posts: 200
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Lol, you just need to have good forum posts which are popular and they will just popup once or twice a day And they aren't that bad, just get rid of the active users link and your set with 100's of users which look like regular guests, which IMO makes people think the site is more popular than it actually is and makes them want to stay
Reply With Quote
  #40  
Old 05-30-2008, 02:31 PM
Alfa1's Avatar
Alfa1 Alfa1 is offline
 
Join Date: Dec 2005
Location: Netherlands
Posts: 3,537
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
This is getting a bit insane now. I am regularly getting over 2100 yahoo slurp spiders at the same time. My server can take it, but I am worried how much this is costing me. I am aware that I can delay the indexing by slurp spiders, but I do not see reason to do this. If there is anyone who can explain this,then that would be very helpful.

At the same time, I do not want to suppress Yahoo! from getting my site high up in their search index. Does anybody know what to do with this?
Reply With Quote
Reply
Page 4 of 5 « First 23 4 5

« Previous Thread | Next Thread »

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT. The time now is 10:58 AM.

Contact Us - Archive - Top

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.04518 seconds
  • Memory Usage 2,247KB
  • Queries Executed 11 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (1)ad_showthread_firstpost
  • (1)ad_showthread_firstpost_sig
  • (1)ad_showthread_firstpost_start
  • (1)bbcode_code
  • (1)bbcode_quote
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)navbar
  • (3)navbar_link
  • (120)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (3)pagenav_pagelink
  • (10)post_thanks_box
  • (10)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (10)post_thanks_postbit_info
  • (10)postbit
  • (10)postbit_onlinestatus
  • (10)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 
Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • fetch_musername
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • pagenav_page
  • pagenav_complete
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete