Miscellaneous Hacks - Ban Spiders by User Agent

What this mod does
With this mod you can enter User Agents to watch or ban, you can also recieve emails or have an Output.txt created and updated with time and date of visits. It doesn't just have to be spiders, you can watch, log or ban any useragent!

How to install
Simply import the product ban_spider, the mod is active by default but none of the other options are turned on.

What is a UserAgent?
http://en.wikipedia.org/wiki/User_agent

Understanding a UserAgent string
http://user-agent-string.info/parse

Genuine User Getting Blocked?
https://vborg.vbsupport.ru/showpost....&postcount=105

Tools to help
http://whatsmyuseragent.com/SwitchingUserAgents.asp
http://www.botsvsbrowsers.com/SimulateUserAgent.asp

FAQ
https://vborg.vbsupport.ru/showpost....&postcount=137

How does it work?
https://vborg.vbsupport.ru/showpost....&postcount=381

What's a bot?
http://en.wikipedia.org/wiki/Spambot

How do i ban a bot?
https://vborg.vbsupport.ru/showpost....&postcount=318
https://vborg.vbsupport.ru/showpost....7&postcount=51

Where's output.txt located?
https://vborg.vbsupport.ru/showpost....&postcount=216

Bad bot lists
https://vborg.vbsupport.ru/showpost....&postcount=259
https://vborg.vbsupport.ru/showpost....&postcount=224
https://vborg.vbsupport.ru/showpost....&postcount=281

Tested on vb3.7.x, vB3.8.x , vB4.x.x but should work on any version.

__________________________________________________ __________________
Special thanks to:
Lior
KH99
BoP5
for helping me sort out a few issues

...and beta testers
ForceHSS (Special thanks to Force for latest testing)
ozzy47
GreyHost

If you use this please mark as INSTALLED

History
9th June 2011 Orginal xml added
12th June 2011 Added both email notification and text file logging
22nd June 2011 Version 2.0.0, Added create thread on activity

1. Added match facility you can now use something like Yandex and it will match MOZILLA/5.0 (COMPATIBLE; YANDEXBOT/3.0; +HTTP://YANDEX.COM/BOTS)
2. Added clickable link to visited thread

22nd September 2011 added user redirect url selection
08th October Beta testing started for thread creation.
20th October Beta testing started for emailing.
21st October Beta testing complete Ver 3.0.0 uploaded
29th October minor fix added to cope with empty userid on thread creation
30th October Beta testing automatic redirection to spiders/bots IP
31st October New xml uploaded with automatic redirect to IP
25th November Minor fix for blank forumid fixed
26th November 2011 Fixed version check & create thread Off by default
17th December 2014 Version 3.1.0 uploaded, Hook changed extra logging and statistics added by Ozzy47 (Chris)
18th December 2014 Version 3.1.1 uploaded, prevented spiders being counted when mod turned off.
17th December 2014 Version 3.1.2 uploaded, due to rogue code from another mod
The Bad Bots list is now included in the product
Please prune out all those that you wish to be able to see your site (i suggest you definately prune out "DA" and "Custo" :

Support will now only be given to those who have this mod marked as INSTALLED

[stator]

Originally Posted by Simon Lloyd

You don't have this marked as installed so i have no idea of which version you downloaded!

Sorry, I marked it.
I've the vB4 version

[Simon Lloyd]

Originally Posted by stator

Sorry, I marked it.
I've the vB4 version

Good because this is the vb4 version thread

[wizardrule]

Great job got rid of all my spiders. Love this, can not beat it for account security, Just a note about adding email in the notification space. I got flooded with emails of spiders being banned. it was truly amazing to see my notification risng up the screen like building blocks.

[Simon Lloyd]

Glad you like it, there is a warning right there near the notifications about the possibility of hundreds of mails, if you are on shared hosting the chances are you are limited to 500 emails per day with a max of 500 per hour, dedicated servers usually get around 5000 per day max of 500 per hour, so using the email notification isn't the best option, in my opinion, and i only added all that notification stuff because people asked for it, the best one to use is the output file, you can prune it heavily at your leisure or just let it grow the only limit you then have is the amount of HDD space you have

[voglermc]

what url are most of you using to redirect spiders to?

[Simon Lloyd]

There's one included in the product but there's nothing to stop you createing a shtml for a 301 and redirecting them there, what i would say as previous is do not redirect them to Google ...etc

[voglermc]

I removed that one and would rather redirect them away from my site altogether. Just wanted to know what a good one would be

[Simon Lloyd]

Ok, in the next few days i'll release another beta where you can either enter a uel to redirect to or redirect them back to their own ip address - anyone think that would be a good idea?

[voglermc]

Hell Yeah!

[Alibass]

Installed and voted MOTM

I have a question, I've got a bot that's been sitting on my site for days.

It's UA is Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; .NET CLR 2.0; and it shows in WOL as hosted-by.leaseweb.com
How can I get rid of this one other than blocking IP?