vb.org Archive - Banning bots that don't follow robots.txt WITHOUT a mod!

vb.org Archive (https://vborg.vbsupport.ru/index.php)

- Forum and Server Management (https://vborg.vbsupport.ru/forumdisplay.php?f=232)

- - Banning bots that don't follow robots.txt WITHOUT a mod! (https://vborg.vbsupport.ru/showthread.php?t=271242)

Simon Lloyd

10-01-2011 03:18 PM

Banning bots that don't follow robots.txt WITHOUT a mod!

Hi all i posted this at vb.com in response to someone but thought it should also be found here :)

Quote:

Originally Posted by Simon Lloyd @ vB.com

Just thought i should post this for all those that don't want to add a mod to do it for you:
if you have mod_write on your apache enabled, add this to your .HTACCESS :

HTML Code:

RewriteEngine On RewriteCond %{HTTP_USER_AGENT} ^baiduspider [NC] RewriteRule .* - [F]

or add this to your httpd.conf

HTML Code:

SetEnvIfNoCase User-Agent "^Baidu" bad_bot

<Directory />

 Order Allow,Deny 

Allow from all 

Deny from env=bad_bot 

</Directory>

souperman

10-04-2011 07:16 AM

Good to know :)

All times are GMT. The time now is 08:37 PM.

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.01016 seconds Memory Usage 1,719KB Queries Executed 10 (?)
More Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (2)bbcode_html_printable (1)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)post_thanks_navbar_search (1)printthread (2)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages: