If you're getting that many blocked...
Look up each IP to find its IP range, then check to see if that range is a hosting company (ie, server farm where no legit users should be coming from). Block the range at the firewall level or using DENY in .htaccess. 2-3 weeks of doing this an the number of blocks you should be seeing will be a handful per day. The bonus is it will block a lot of other bad crawlers, decrease bandwidth the server users and result in faster site response time.
|