The Arcive of Official vBulletin Modifications Site.

SPider question · **Released**: 08-26-2003

A few days ago, I got "visited" by a spider that used 1Gb of bandwidth! My Control Panel shows this:

Unknown robot (identified by 'crawl')
HITS: 43632
Bandwidth: 1007.05 MB

I have changed my board so that unregistered users cannot view, and I added a robots.txt file.

Is there anything else I should do?

How can I find out what spider this one is?

Should the robots.txt file be in "public_html", or "public_html/forums".

Thanks

:-jon

lasto · #2 08-27-2003, 12:35 AM

should be in main root im sure of it

also is this what is in your robot file

# go away
User-agent: *
Disallow: /

as this indicates that no robots should visit this site further

1gb of your bandwidth is bad esp for a robot you dont want in the first place.

Instead of us blocking em it should be if there is no file there saying allow they should move on.

filburt1 · #3 08-27-2003, 01:38 AM

<a href="https://vborg.vbsupport.ru/showthread.php?s=&threadid=55092" target="_blank">https://vborg.vbsupport.ru/showt...threadid=55092</a> !

Erwin · #4 08-27-2003, 04:40 AM

Don't block all spiders!

You may only want to block specific ones from smaller less significant search engines.

lasto · #5 08-27-2003, 01:01 PM

how about a list of spiders to block then or ones that seem to want to use all your bandwidth as to be honest i aint got a clue what spidering and wot aint.

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.04026 seconds Memory Usage 2,229KB Queries Executed 18 (?)
More Information
Template Usage: (1)SHOWTHREAD (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (1)ad_showthread_beforeqr (1)footer (1)forumjump (1)forumrules (1)gobutton (1)header (1)headinclude (1)modsystem_post (1)navbar (6)navbar_link (120)option (5)post_thanks_box (5)post_thanks_button (1)post_thanks_javascript (1)post_thanks_navbar_search (5)post_thanks_postbit_info (4)postbit (5)postbit_onlinestatus (5)postbit_wrapper (1)spacer_close (1)spacer_open (1)tagbit_wrapper Phrase Groups Available: global inlinemod postbit posting reputationlevel showthread	Included Files: ./showthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/functions_bigthree.php ./includes/class_postbit.php ./includes/class_bbcode.php ./includes/functions_reputation.php ./includes/functions_post_thanks.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete showthread_start showthread_getinfo forumjump showthread_post_start showthread_query_postids showthread_query bbcode_fetch_tags bbcode_create showthread_postbit_create postbit_factory postbit_display_start post_thanks_function_post_thanks_off_start post_thanks_function_post_thanks_off_end post_thanks_function_fetch_thanks_start post_thanks_function_fetch_thanks_end post_thanks_function_thanked_already_start post_thanks_function_thanked_already_end fetch_musername postbit_imicons bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete postbit_display_complete post_thanks_function_can_thank_this_post_start tag_fetchbit_complete forumrules navbits navbits_complete showthread_complete
Messages:

The Arcive of Official vBulletin Modifications Site.

It is not a VB3 engine, just a parsed copy!