Go Back   vb.org Archive > vBulletin 3 Discussion > vB3 General Discussions
Reload this Page 549 spiders on my site
FAQ Community Calendar Today's Posts Search

Reply
Page 1 of 2 1 2
 
Thread Tools Display Modes
  #1  
Old 01-19-2009, 03:29 PM
dtv100 dtv100 is offline
 
Join Date: Apr 2007
Location: in the south of the north
Posts: 307
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default 549 spiders on my site
549 spiders at the same time is that a good thing or bad ?

my server load is 1.46 because of this .
Reply With Quote
  #2  
Old 01-19-2009, 03:42 PM
snakes1100 snakes1100 is offline
 
Join Date: Dec 2001
Location: Michigan
Posts: 3,733
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Use a robots.txt file to limit the bots, i wont hazard a guess, but i would say Yahoo is visiting you today.

Typically i block all but one of the Yahoo's bot server IP's until they start following the robots.txt file you most likely didn't upload to your site.
Reply With Quote
  #3  
Old 01-19-2009, 07:35 PM
dtv100 dtv100 is offline
 
Join Date: Apr 2007
Location: in the south of the north
Posts: 307
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
most of then gone now. right now i have the average i always have between 10 to 30 spiders and is much better.
Reply With Quote
  #4  
Old 01-19-2009, 08:59 PM
edermix edermix is offline
 
Join Date: Aug 2008
Posts: 87
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
How did you get so many spiders? How was your robots.txt file? Do you have a mod so that you did the visits of so many spiders?

I would much all these visits from spiders in my forum. Help me.
Reply With Quote
  #5  
Old 01-19-2009, 11:53 PM
glennybee glennybee is offline
 
Join Date: Feb 2008
Location: Scotland
Posts: 329
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Why would you want to restrict spiders? It's spiders that get you organic traffic.
Reply With Quote
  #6  
Old 01-20-2009, 12:07 AM
nexialys
Guest
  
Posts: n/a
Default
two different spiders, two different presences on your site:

1- Googlebot: a single bot which visit each link of your site page by page
2- Yahoobot: a pyramidal bot which will call one friend per new link found on a page

Google will appear once, on a long period because it visit the site on its own... if you see multiple google bots, they usually are driven by references (too much pages to see, would be too slow if a single bot)

Yahoo bot will have so much links to visit that your sessions are overloaded like if there is really millions of visitors, but it's just a cascade of different IP used by a single engine.
Reply With Quote
  #7  
Old 01-20-2009, 01:23 AM
dtv100 dtv100 is offline
 
Join Date: Apr 2007
Location: in the south of the north
Posts: 307
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Quote:
Originally Posted by edermix View Post
How did you get so many spiders? How was your robots.txt file? Do you have a mod so that you did the visits of so many spiders?

I would much all these visits from spiders in my forum. Help me.
i not using no hack for this .
usually i just get from 10 to 30 spiders a hour don't know what happen today .
Reply With Quote
  #8  
Old 01-22-2009, 09:50 PM
edermix edermix is offline
 
Join Date: Aug 2008
Posts: 87
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
My robots. See:

http://www.gcert.com.br/robots.txt

3 spiders.
Reply With Quote
  #9  
Old 02-08-2009, 11:55 AM
edermix edermix is offline
 
Join Date: Aug 2008
Posts: 87
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Help-me Somewere??
Reply With Quote
  #10  
Old 02-08-2009, 10:43 PM
kNeeLy's Avatar
kNeeLy kNeeLy is offline
 
Join Date: Nov 2008
Location: ohio
Posts: 244
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
yea..if sumone could explain the whole spider thing...lol...I know my server will make me a robot.txt file or w/e it is..but i have no idea how to use it...can anyone help? :-D
Reply With Quote
Reply
Page 1 of 2 1 2

« Previous Thread | Next Thread »

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT. The time now is 05:20 AM.

Contact Us - Archive - Top

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.12213 seconds
  • Memory Usage 2,246KB
  • Queries Executed 13 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (1)ad_showthread_firstpost
  • (1)ad_showthread_firstpost_sig
  • (1)ad_showthread_firstpost_start
  • (1)bbcode_quote
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)navbar
  • (3)navbar_link
  • (120)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (1)pagenav_pagelink
  • (10)post_thanks_box
  • (10)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (10)post_thanks_postbit_info
  • (10)postbit
  • (9)postbit_onlinestatus
  • (10)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 
Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_postinfo_query
  • fetch_postinfo
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • fetch_musername
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • pagenav_page
  • pagenav_complete
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete