The Arcive of Official vBulletin Modifications Site.

paldo · #1 11-13-2006, 03:41 PM

this beats anything i have ever seen yahoo spiders have a love affair with my forums most people are happy to have a spider or 3 but i have had as many as 600 yahoo spiders on my board at once .. i have played with a robots.txt and the only way i can slow them down is by disallowing access to my forums which defeats the purpose of having spiders on your forums when they cant read it .. i dont mind them being there but a few would be nice rather then by the 100's

Currently Active Users: 240 (9 members and 231 guests)

sinse i posted this and refreshed my page here is the new count ..

Visitors (27), Google Spiders (1), Yahoo! Slurp Spiders (504)
its insane

Zachery · #2 11-13-2006, 03:48 PM

Give them a day or two and they wil wonder off.

paldo · #3 11-13-2006, 03:50 PM

thats what i thought this has been a battle for over a month .my site has 200-300 new post a day but this is nuts lol

Currently Active Users: 342 (9 members and 333 guests)

i now see this site has alot of yahoo bots online as well i just dont see why they hafta crawl in masses like that

SaN-DeeP · #4 11-13-2006, 10:37 PM

check the robots.txt file for this site -> vbulletin.com/forums/robots.txt
you can setup a crawl delay.

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.03304 seconds Memory Usage 2,178KB Queries Executed 11 (?)
More Information
Template Usage: (1)SHOWTHREAD (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (1)ad_showthread_beforeqr (1)ad_showthread_firstpost (1)ad_showthread_firstpost_sig (1)ad_showthread_firstpost_start (1)footer (1)forumjump (1)forumrules (1)gobutton (1)header (1)headinclude (1)navbar (3)navbar_link (120)option (4)post_thanks_box (4)post_thanks_button (1)post_thanks_javascript (1)post_thanks_navbar_search (4)post_thanks_postbit_info (4)postbit (4)postbit_onlinestatus (4)postbit_wrapper (1)spacer_close (1)spacer_open (1)tagbit_wrapper Phrase Groups Available: global inlinemod postbit posting reputationlevel showthread	Included Files: ./showthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/functions_bigthree.php ./includes/class_postbit.php ./includes/class_bbcode.php ./includes/functions_reputation.php ./includes/functions_post_thanks.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete showthread_start showthread_getinfo forumjump showthread_post_start showthread_query_postids showthread_query bbcode_fetch_tags bbcode_create showthread_postbit_create postbit_factory postbit_display_start post_thanks_function_post_thanks_off_start post_thanks_function_post_thanks_off_end post_thanks_function_fetch_thanks_start post_thanks_function_fetch_thanks_end post_thanks_function_thanked_already_start post_thanks_function_thanked_already_end fetch_musername postbit_imicons bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete postbit_display_complete post_thanks_function_can_thank_this_post_start tag_fetchbit_complete forumrules navbits navbits_complete showthread_complete
Messages:

The Arcive of Official vBulletin Modifications Site.

It is not a VB3 engine, just a parsed copy!