The Arcive of Official vBulletin Modifications Site.

smoknz28 · #1 10-05-2006, 12:20 AM

vB 3.6.1

I don't understand fully about what these 'spiders' are about however, I would like to reduce how many of them hit my site.

Today I had 120+ of them at one time viewing one of my forums. What I do understand is that these can slow down my site.

Is there a way that I can stop these or limit these spiders such as Yahoo! Slurp Spider from hitting my site?

Thanks gang,
Mark

www.f-bodyhideout.com

letsjoy · #2 10-05-2006, 01:00 AM

yes u can ban al spiders... but doin that will not ur pages cralwed in SE's

raiderlax · #3 10-05-2006, 01:16 AM

how do you get more spiders to go on your site?

Spiffware · #4 10-05-2006, 01:21 AM

Quote:

Originally Posted by raiderlax

how do you get more spiders to go on your site?

im with you i think its in the right keywords...

Exitilus · #5 10-05-2006, 03:36 AM

You submit your site to the Search Site (ie Google & Yahoo)

TheBlackPoet · #6 10-28-2006, 04:45 PM

i have always liked a few yahoo spiders on my site crawling... (at least when i learned they help with SEO's)... but this weekend, i was bombarded with 100 then 150... when i got up to 185 spiders, i knew that was a bit much... so now I'm asking, do i really want that uninvited intrusion on my site? even a good thing should have permission to be good...

my five-minute search lead me to making a robots.txt page..... i collected a few lines of disallow texts to block spiders... i dont know if this will work, but until someone smarter comes up with something better, i havent had a new spider since i uploaded this...

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.03700 seconds Memory Usage 2,219KB Queries Executed 14 (?)
More Information
Template Usage: (1)SHOWTHREAD (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (1)ad_showthread_beforeqr (1)ad_showthread_firstpost (1)ad_showthread_firstpost_sig (1)ad_showthread_firstpost_start (1)bbcode_quote (1)footer (1)forumjump (1)forumrules (1)gobutton (1)header (1)headinclude (1)navbar (3)navbar_link (120)option (6)post_thanks_box (6)post_thanks_button (1)post_thanks_javascript (1)post_thanks_navbar_search (6)post_thanks_postbit_info (6)postbit (1)postbit_attachment (6)postbit_onlinestatus (6)postbit_wrapper (1)spacer_close (1)spacer_open (1)tagbit_wrapper Phrase Groups Available: global inlinemod postbit posting reputationlevel showthread	Included Files: ./showthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/functions_bigthree.php ./includes/class_postbit.php ./includes/class_bbcode.php ./includes/functions_reputation.php ./includes/functions_post_thanks.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_postinfo_query fetch_postinfo fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete showthread_start showthread_getinfo forumjump showthread_post_start showthread_query_postids showthread_query bbcode_fetch_tags bbcode_create showthread_postbit_create postbit_factory postbit_display_start post_thanks_function_post_thanks_off_start post_thanks_function_post_thanks_off_end post_thanks_function_fetch_thanks_start post_thanks_function_fetch_thanks_end post_thanks_function_thanked_already_start post_thanks_function_thanked_already_end fetch_musername postbit_imicons bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete postbit_display_complete post_thanks_function_can_thank_this_post_start postbit_attachment tag_fetchbit_complete forumrules navbits navbits_complete showthread_complete
Messages:

The Arcive of Official vBulletin Modifications Site.

It is not a VB3 engine, just a parsed copy!