Miscellaneous Hacks - Ban Spiders by User Agent

What this mod does
With this mod you can enter User Agents to watch or ban, you can also recieve emails or have an Output.txt created and updated with time and date of visits. It doesn't just have to be spiders, you can watch, log or ban any useragent!

How to install
Simply import the product ban_spider, the mod is active by default but none of the other options are turned on.

What is a UserAgent?
http://en.wikipedia.org/wiki/User_agent

Understanding a UserAgent string
http://user-agent-string.info/parse

Genuine User Getting Blocked?
https://vborg.vbsupport.ru/showpost....&postcount=105

Tools to help
http://whatsmyuseragent.com/SwitchingUserAgents.asp
http://www.botsvsbrowsers.com/SimulateUserAgent.asp

FAQ
https://vborg.vbsupport.ru/showpost....&postcount=137

How does it work?
https://vborg.vbsupport.ru/showpost....&postcount=381

What's a bot?
http://en.wikipedia.org/wiki/Spambot

How do i ban a bot?
https://vborg.vbsupport.ru/showpost....&postcount=318
https://vborg.vbsupport.ru/showpost....7&postcount=51

Where's output.txt located?
https://vborg.vbsupport.ru/showpost....&postcount=216

Bad bot lists
https://vborg.vbsupport.ru/showpost....&postcount=259
https://vborg.vbsupport.ru/showpost....&postcount=224
https://vborg.vbsupport.ru/showpost....&postcount=281

Tested on vb3.7.x, vB3.8.x , vB4.x.x but should work on any version.

__________________________________________________ __________________
Special thanks to:
Lior
KH99
BoP5
for helping me sort out a few issues

...and beta testers
ForceHSS (Special thanks to Force for latest testing)
ozzy47
GreyHost

If you use this please mark as INSTALLED

History
9th June 2011 Orginal xml added
12th June 2011 Added both email notification and text file logging
22nd June 2011 Version 2.0.0, Added create thread on activity

1. Added match facility you can now use something like Yandex and it will match MOZILLA/5.0 (COMPATIBLE; YANDEXBOT/3.0; +HTTP://YANDEX.COM/BOTS)
2. Added clickable link to visited thread

22nd September 2011 added user redirect url selection
08th October Beta testing started for thread creation.
20th October Beta testing started for emailing.
21st October Beta testing complete Ver 3.0.0 uploaded
29th October minor fix added to cope with empty userid on thread creation
30th October Beta testing automatic redirection to spiders/bots IP
31st October New xml uploaded with automatic redirect to IP
25th November Minor fix for blank forumid fixed
26th November 2011 Fixed version check & create thread Off by default
17th December 2014 Version 3.1.0 uploaded, Hook changed extra logging and statistics added by Ozzy47 (Chris)
18th December 2014 Version 3.1.1 uploaded, prevented spiders being counted when mod turned off.
17th December 2014 Version 3.1.2 uploaded, due to rogue code from another mod
The Bad Bots list is now included in the product
Please prune out all those that you wish to be able to see your site (i suggest you definately prune out "DA" and "Custo" :

Support will now only be given to those who have this mod marked as INSTALLED

[CAG CheechDogg]

Crazy how a leading or trailing space can jack things up lol .. I remember trying to block facebook completely when I first installed your mod here Simon and it wasn't working ..why? the damn trailing space!! lol

[Gadget_Guy]

Sorry Simon,

I meant no disrespect... I thought you guys were working together.

My apologies.

Could I send you my list via PM and a screenshot of what I am seeing in terms of the spiders appearing?

D.

[ozzy47]

Originally Posted by Simon Lloyd

Hey Gadget Guy, no disrespect but the mod here is mine and isn't marked as reusable code, Ozzy may post what he's tried or done but wont necessarily be added to this mod, however Ozzy has developed one like this with other measures, you can get it at his site.

If you are being hit by spiders with this mod in place it will be because there is an anomaly in your list, this list isn't exhaustive but here's a few reasons why:
Entry in list has a leading or trailing space
Entry has a typo of some sort
Entry doesn't actually represent the bot you think it does (i.e Ahrefsbot I believe has a different name in the UA)
Mod run order may conflict with another mod using the same hook
There are other reasons but those should get you going!

No I have not developed a mod similar to this. I recommend this mod to everyone. If you remember correctly, I was one of your beta testers way back in the day.

My actual intention with what I have done, was to PM it to you first, and get your input, before I said anything to anyone else, or maybe you update the mod with it.

So let me know what you want me to do.

[Alan_SP]

Originally Posted by CAG CheechDogg

You can get it here my Man : https://vborg.vbsupport.ru/showthrea...134.184.0%2F21

Thank you, I tried to like your post, but there's a limit in this.

Anyway, I see you put this in htaccess, I think it's better added to firewall rules for denying hosts. It works much faster. Of course, if you have access to firewall in that way.

[ozzy47]

Yeah, if you have a dedi, or a vps, you would add it to the firewall there, stop them cold.

[Simon Lloyd]

Originally Posted by ozzy47

No I have not developed a mod similar to this. I recommend this mod to everyone. If you remember correctly, I was one of your beta testers way back in the day.

My actual intention with what I have done, was to PM it to you first, and get your input, before I said anything to anyone else, or maybe you update the mod with it.

So let me know what you want me to do.

I thought you had developed something that's why planned improvements for this have been shelved (no point in duplication ), in that case Ozzy PM away

[Simon Lloyd]

Originally Posted by Gadget_Guy

Sorry Simon,

I meant no disrespect... I thought you guys were working together.

My apologies.

Could I send you my list via PM and a screenshot of what I am seeing in terms of the spiders appearing?



D.

Yes of course

[Max Taxable]

Originally Posted by Simon Lloyd

I thought you had developed something that's why planned improvements for this have been shelved (no point in duplication ), in that case Ozzy PM away

No sir we have been trying to solve the mystery of why Baidu gets through on some v4 installations, but not all and never a v3, and my hook conflict idea opened a new can of worms for investigation, and Ozz found something very interesting.

[ForceHSS]

Originally Posted by Max Taxable

No sir we have been trying to solve the mystery of why Baidu gets through on some v4 installations, but not all and never a v3, and my hook conflict idea opened a new can of worms for investigation, and Ozz found something very interesting.

What interesting thing was found

[CAG CheechDogg]

Yeah ...Yeah .. what "something very interesting" do you speak of .....