Go Back   vb.org Archive > vBulletin 3 Discussion > vB3 General Discussions
FAQ Community Calendar Today's Posts Search

Reply
 
Thread Tools Display Modes
  #1  
Old 07-06-2007, 05:02 AM
Southernphuk's Avatar
Southernphuk Southernphuk is offline
 
Join Date: Aug 2004
Posts: 26
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default So why has my site become popular for spiders, webcralwlers and bots now?

Hola,

I've noticed over the last month that the spiders, bots and webcrawlers have really punched it up a notch, where I used to get around 150-200 a day I'm now getting closer to 2000 of them a day now and I can't for the life of me figure out why!

Any clues or ideas for the clueless here? I've got a robot.txt file but that is to keep them out of certain areas that they really have no business bothering with (profiles, pm's, etc) but have shied away from outright cutting them off from the site as I figure that is a good way to have a presence in the search engines which of course everyone wants them to funnel their way.

Anyhow, extremely curious about this and wouldn't mind some thoughts and insight on this.
Reply With Quote
  #2  
Old 07-06-2007, 05:06 AM
SCRIPT3R SCRIPT3R is offline
 
Join Date: Jan 2005
Posts: 1,303
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

just count yourself lucky and leave it at that.
Reply With Quote
  #3  
Old 07-06-2007, 05:16 AM
Southernphuk's Avatar
Southernphuk Southernphuk is offline
 
Join Date: Aug 2004
Posts: 26
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Hah, believe me, I really don't want to get rid of them as frankly they aren't eating up 'that' much bandwidth (well, the google one ate through a goodly bit last month but still), I'm just curious what triggers something like this.

Regards,
Nathan
Reply With Quote
  #4  
Old 07-06-2007, 04:10 PM
SCRIPT3R SCRIPT3R is offline
 
Join Date: Jan 2005
Posts: 1,303
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

nobody really knows for sure... luck of the draw.
Reply With Quote
  #5  
Old 07-06-2007, 04:14 PM
acertek acertek is offline
 
Join Date: Jun 2007
Location: Dallas
Posts: 26
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Yeah I've noticed this lately too. What I've been noticing is inktomi/yahoo seems to have a spider that appears as 100+ users on your forum and crawls it with multiple legs. I think as your forum grows they put largers spiders on it to crawl it faster.
Reply With Quote
  #6  
Old 07-06-2007, 04:33 PM
Brandon Sheley's Avatar
Brandon Sheley Brandon Sheley is offline
 
Join Date: Mar 2005
Location: Google Kansas
Posts: 4,678
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

back links
Whats your site url ? I'll be able to tell you in a matter on minutes..
Reply With Quote
  #7  
Old 07-15-2007, 01:33 PM
mfyvie mfyvie is offline
 
Join Date: Mar 2007
Location: Zurich, Switzerland
Posts: 336
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally Posted by Southernphuk View Post
Any clues or ideas for the clueless here? I've got a robot.txt file but that is to keep them out of certain areas that they really have no business bothering with (profiles, pm's, etc) but have shied away from outright cutting them off from the site as I figure that is a good way to have a presence in the search engines which of course everyone wants them to funnel their way.

Anyhow, extremely curious about this and wouldn't mind some thoughts and insight on this.
Most of the evil ones won't respect robots.txt anyway. You should care about them though - the nastier ones are just stealing your content and bringing you nothing in return. At worst they can bring your site to a standstill if they are poorly written (I've had this happen to me on a few occassions). I block a big list of the bad ones at my webserver and my forum response time increased immediately. I still allow all search engines through unless they are specifically blocked (I think I block around 200).

I also wrote a small mod to remove spiders from the statistics on your forum. This doesn't block anything, but it is useful to get rid of the 100's of sessions started by spiders like Yahoo's slurp.
Reply With Quote
Reply


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 02:24 AM.


Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.03526 seconds
  • Memory Usage 2,214KB
  • Queries Executed 13 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (1)ad_showthread_firstpost
  • (1)ad_showthread_firstpost_sig
  • (1)ad_showthread_firstpost_start
  • (1)bbcode_quote
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)navbar
  • (3)navbar_link
  • (120)option
  • (7)post_thanks_box
  • (7)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (7)post_thanks_postbit_info
  • (7)postbit
  • (7)postbit_onlinestatus
  • (7)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 

Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_postinfo_query
  • fetch_postinfo
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • fetch_musername
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete