Go Back   vb.org Archive > vBulletin 4 Discussion > vB4 General Discussions
FAQ Community Calendar Today's Posts Search

Reply
 
Thread Tools Display Modes
  #1  
Old 02-28-2011, 06:12 PM
Baf_Jams Baf_Jams is offline
 
Join Date: Mar 2008
Location: Derby UK
Posts: 106
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default is it possible to Limit Spiders ?

Hi all
I have over 400 spiders on my website for 2 weeks now and yes i like them to index my website is there any text that i could put in robot.txt to limit the amount that are on my website ?
I have spent too long now on google searching and not getting anywhere
as you can see at one point there was over 600, we are a small gaming squad been around since 2001 and have about 30-40 active members
Reply With Quote
  #2  
Old 02-28-2011, 06:17 PM
your24hourstore your24hourstore is offline
 
Join Date: Feb 2010
Posts: 1,226
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

probably htaccess but not robots.txt it allows or Denys only wont specify how many
but I cant do it i hate htaccess
Reply With Quote
  #3  
Old 03-04-2011, 12:37 AM
Videx's Avatar
Videx Videx is offline
 
Join Date: Feb 2007
Posts: 3,085
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

First you need to figure out where all those guests are coming from. Chances are they aren't spiders. Check your referrers in cpanel.
Reply With Quote
  #4  
Old 03-04-2011, 12:39 AM
Boofo's Avatar
Boofo Boofo is offline
 
Join Date: Mar 2002
Location: Des Moines, IA (USA)
Posts: 15,776
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

How do you know some of them aren't spiders? What do they show up as in the Who's Online page?
Reply With Quote
  #5  
Old 03-04-2011, 12:55 AM
Videx's Avatar
Videx Videx is offline
 
Join Date: Feb 2007
Posts: 3,085
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally Posted by Boofo View Post
How do you know some of them aren't spiders? What do they show up as in the Who's Online page?
Are you asking me? I've discussed elsewhere about our recent troubles with "Guest Attacks", where 10X the normal number of guests have been logging on and all are doing different things. They show up as Guests, and they're all from different IPs.

However, in our case it's been about two weeks since I uninstalled FeedWind and so far we haven't seen another attack (knock on wood).
Reply With Quote
  #6  
Old 04-12-2011, 09:20 PM
Baf_Jams Baf_Jams is offline
 
Join Date: Mar 2008
Location: Derby UK
Posts: 106
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Sorry for the delay, I just left them alone and they seemed to go away !!!!
and yes they do show up on who's online
I still have about 10 bots online now though but like i say i need them to index me
heres a SS of the bots online now

Reply With Quote
  #7  
Old 04-12-2011, 11:52 PM
Videx's Avatar
Videx Videx is offline
 
Join Date: Feb 2007
Posts: 3,085
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

I wish I could say the same. It's been a couple weeks since I've reset our counter, and it's only showing Most Online at 137 right now. That's better than during the mad Guest Attacks, but still above normal for us.
Reply With Quote
Reply


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 11:26 AM.


Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.05895 seconds
  • Memory Usage 2,213KB
  • Queries Executed 13 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (1)ad_showthread_firstpost
  • (1)ad_showthread_firstpost_sig
  • (1)ad_showthread_firstpost_start
  • (1)bbcode_quote
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)navbar
  • (3)navbar_link
  • (120)option
  • (7)post_thanks_box
  • (7)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (7)post_thanks_postbit_info
  • (7)postbit
  • (7)postbit_onlinestatus
  • (7)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 

Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_postinfo_query
  • fetch_postinfo
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • fetch_musername
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete