The Arcive of vBulletin Modifications Site.

theFAILURE · #1 03-08-2008, 10:16 PM

I have noticed lately that the Yahoo Slurp Bots are hitting hard.

When I see them in mass on my forum, sure enough I come here, and they are in mass as well.

When they die down, sure enough they are died down here too.

Why is that you think?

Right now I got over 250:

#2 03-09-2008, 04:07 AM

Toss this:

Code:

User-agent: Slurp
Crawl-delay: 60

In your robots.txt. I can't believe that after all this time a company as big as Yahoo still hasn't tamed their obnoxious spider. I've had 2k+ instances of it on my site at once.

theFAILURE · #3 03-09-2008, 04:48 AM

Thats the crawl delay, but I do not think it stops the number of spiders at once that crawl.

They need to implement something like:

Spider-max: 20 (which would mean a maximum of 20 spiders at a time)

It puzzles me too why they never fixed this.

SEOvB · #4 03-09-2008, 05:58 AM

your avatar is pretty hilarous

theFAILURE · #5 03-09-2008, 06:05 AM

You should see the full version

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.05946 seconds Memory Usage 4,663KB Queries Executed 11 (?)
More Information
Template Usage: (1)SHOWTHREAD (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (1)ad_showthread_beforeqr (1)ad_showthread_firstpost (1)ad_showthread_firstpost_sig (1)ad_showthread_firstpost_start (1)bbcode_code (1)footer (1)forumjump (1)forumrules (1)gobutton (1)header (1)headinclude (1)navbar (3)navbar_link (120)option (5)post_thanks_box (5)post_thanks_button (1)post_thanks_javascript (1)post_thanks_navbar_search (5)post_thanks_postbit_info (5)postbit (4)postbit_onlinestatus (5)postbit_wrapper (1)spacer_close (1)spacer_open (1)tagbit_wrapper Phrase Groups Available: global inlinemod postbit posting reputationlevel showthread	Included Files: ./showthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/functions_bigthree.php ./includes/class_postbit.php ./includes/class_bbcode.php ./includes/functions_reputation.php ./includes/functions_post_thanks.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete showthread_start showthread_getinfo forumjump showthread_post_start showthread_query_postids showthread_query bbcode_fetch_tags bbcode_create showthread_postbit_create postbit_factory postbit_display_start post_thanks_function_post_thanks_off_start post_thanks_function_post_thanks_off_end post_thanks_function_fetch_thanks_start post_thanks_function_fetch_thanks_end post_thanks_function_thanked_already_start post_thanks_function_thanked_already_end fetch_musername postbit_imicons bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete postbit_display_complete post_thanks_function_can_thank_this_post_start tag_fetchbit_complete forumrules navbits navbits_complete showthread_complete
Messages: