Thread: Miscellaneous Hacks - Ban Spiders by User Agent
View Single Post
  #447  
Old 12-15-2012, 09:45 PM
My Hattiesburg My Hattiesburg is offline
 
Join Date: Jun 2012
Posts: 128
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Okay, I've installed this and it seems to be working fine, but I have some questions.

At first the Baidu spider was hammering us, hitting the site about every 8 seconds, but now it seems to have more or less given up on us. Yandex, on the other hand, seems to have intensified it's attempted crawls. At first it was hitting the site about every 25 seconds but over the course of this install has ranged everywhere from every 1 second to where it's at now, about every 1.5 minutes. The 1.5 minute attempts have only occurred in the last couple of days.

A couple of days ago when Yandex was hitting us every 1 second, we had some server load issues. I don't know if this is related but it seems it might be and I'm wondering if logging the blocks to a text file might be counterproductive in that area, writing to the file every second or so.

Also, Yandex is using the same IP address every time, so I thought it might be best to just block it using the .htaccess file, but that doesn't seem to have had any effect. Is this mod redirecting Yandex before it has a chance to read the .htaccess file or is Yandex simply ignoring it?
Reply With Quote
 
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01068 seconds
  • Memory Usage 1,760KB
  • Queries Executed 11 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD_SHOWPOST
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)post_thanks_box
  • (1)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (1)post_thanks_postbit_info
  • (1)postbit
  • (1)postbit_onlinestatus
  • (1)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • reputationlevel
  • showthread
Included Files:
  • ./showpost.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_postinfo_query
  • fetch_postinfo
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showpost_start
  • bbcode_fetch_tags
  • bbcode_create
  • postbit_factory
  • showpost_post
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • fetch_musername
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • showpost_complete