The Arcive of Official vBulletin Modifications Site.

Spider/Bot Assault in progress · **Released**: 07-17-2003

Hello all,

This is my first post to these forums, Usualy I read this massive library, and find the answers I need. However yesterday, I was looking ofver my forum and I have found that a massive spider or Bot assault is going on at my website. I have traced the bouts IP back to inktomisearch.com which is a search engine that suposedly got bought out by Yahoo.

I have had up to 44 Spider/bot visitors who are I assume "web crawling" my site. I use the Vbportal v2.3.0 (Final) and I am currently using vbulletin V2.3.0.

My question is how can I dump these little spiders as they have been at my site for 2 days now, is there a hack to loose these bots, or should I not even bother with it?

Just looking for suggestions or methods really!

Take care all, you guys Rock!
Pinpoint2RHh
http://www.2RHh.com

Oblivion Knight · #2 07-17-2003, 03:05 PM

I've had these bots crawling Umbrella Online since yesterday afternoon..
I wouldn't know why you'd want to get rid of them though, search engine bots crawling more of your site = more "results" located by a search with Yahoo! or other search engines = more traffic to your forums..

Dan · #3 07-17-2003, 03:36 PM

search for robot.txt on the net and it will tell you how to stop them

allan grossman · #4 07-18-2003, 02:03 AM

There are several ways to keep the spiders out if you want to.

You can use a meta tag on any page to keep robots from crawling that page or any links they find on it.

<meta name="robots" content="noindex, nofollow">

You can place a text file called "robots.txt" in the web root (and only in the web root) that says

User-agent: inktomisearch.com
Disallow: /

or you can use an .htaccess file in any directory you don't want them in - like this:

Order allow,deny
allow from all
deny from inktomisearch.com

Notice there is no space between allow and deny - that's important. You can also use this in httpd.conf if you're running Apache.

But - if you'd prefer not to do DNS lookups for everyone that hits your site you can do it like this -

Order allow,deny
allow from all
deny from 209.131.63

but you'd substitute the subnet the spiders are coming from.

Pinpoint2RHh · #5 07-18-2003, 03:34 PM

Thanks so much Allan thats perfect info!

I use Apache, So I think it would be best to just refuse the IP block method.

Thanks again, I was looking over the text for the Robots, that Oricon pointed out. (Thanks to you too Oricon)

Did I mention that you folks rock! :cheeky:

Pinpoint2RHh

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.04655 seconds Memory Usage 2,238KB Queries Executed 20 (?)
More Information
Template Usage: (1)SHOWTHREAD (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (1)ad_showthread_beforeqr (1)footer (1)forumjump (1)forumrules (1)gobutton (1)header (1)headinclude (1)modsystem_post (1)navbar (6)navbar_link (120)option (5)post_thanks_box (5)post_thanks_button (1)post_thanks_javascript (1)post_thanks_navbar_search (5)post_thanks_postbit_info (4)postbit (5)postbit_onlinestatus (5)postbit_wrapper (1)spacer_close (1)spacer_open (1)tagbit_wrapper Phrase Groups Available: global inlinemod postbit posting reputationlevel showthread	Included Files: ./showthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/functions_bigthree.php ./includes/class_postbit.php ./includes/class_bbcode.php ./includes/functions_reputation.php ./includes/functions_post_thanks.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_postinfo_query fetch_postinfo fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete showthread_start showthread_getinfo forumjump showthread_post_start showthread_query_postids showthread_query bbcode_fetch_tags bbcode_create showthread_postbit_create postbit_factory postbit_display_start post_thanks_function_post_thanks_off_start post_thanks_function_post_thanks_off_end post_thanks_function_fetch_thanks_start post_thanks_function_fetch_thanks_end post_thanks_function_thanked_already_start post_thanks_function_thanked_already_end fetch_musername postbit_imicons bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete postbit_display_complete post_thanks_function_can_thank_this_post_start tag_fetchbit_complete forumrules navbits navbits_complete showthread_complete
Messages:

The Arcive of Official vBulletin Modifications Site.

It is not a VB3 engine, just a parsed copy!