The Arcive of vBulletin Modifications Site.

#1 02-21-2006, 03:58 PM

Just looking for any advice here. Spiders are dominating the MySQL process on my box - at any given time I have 15-20 different spiders online, and at times it doubles my online user/query count.

Is there a decent way to lessen their load, while at the same time keeping the "benefits" of having them crawl my site? I really don't want to bump up my monthly server cost to accomodate engine rankings.

Thanks in advance.

Reeve of shinra · #2 02-21-2006, 06:04 PM

<a href="http://www.theadminzone.com/forums/showthread.php?t=19465" target="_blank">http://www.theadminzone.com/forums/s...ad.php?t=19465</a>

calorie · #3 02-21-2006, 06:46 PM

Oldie but might still be goodie: here

S@NL - BlackBik · #4 02-21-2006, 06:50 PM

Check the robots.txt tutorial.

#5 02-22-2006, 04:39 PM

Great info - thanks very much all.

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.06545 seconds Memory Usage 4,658KB Queries Executed 11 (?)
More Information
Template Usage: (1)SHOWTHREAD (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (1)ad_showthread_beforeqr (1)ad_showthread_firstpost (1)ad_showthread_firstpost_sig (1)ad_showthread_firstpost_start (1)footer (1)forumjump (1)forumrules (1)gobutton (1)header (1)headinclude (1)navbar (3)navbar_link (120)option (5)post_thanks_box (5)post_thanks_button (1)post_thanks_javascript (1)post_thanks_navbar_search (5)post_thanks_postbit_info (5)postbit (3)postbit_onlinestatus (5)postbit_wrapper (1)spacer_close (1)spacer_open (1)tagbit_wrapper Phrase Groups Available: global inlinemod postbit posting reputationlevel showthread	Included Files: ./showthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/functions_bigthree.php ./includes/class_postbit.php ./includes/class_bbcode.php ./includes/functions_reputation.php ./includes/functions_post_thanks.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete showthread_start showthread_getinfo forumjump showthread_post_start showthread_query_postids showthread_query bbcode_fetch_tags bbcode_create showthread_postbit_create postbit_factory postbit_display_start post_thanks_function_post_thanks_off_start post_thanks_function_post_thanks_off_end post_thanks_function_fetch_thanks_start post_thanks_function_fetch_thanks_end post_thanks_function_thanked_already_start post_thanks_function_thanked_already_end fetch_musername bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete postbit_display_complete post_thanks_function_can_thank_this_post_start postbit_imicons tag_fetchbit_complete forumrules navbits navbits_complete showthread_complete
Messages: