Miscellaneous Hacks - Ban Spiders by User Agent

What this mod does
With this mod you can enter User Agents to watch or ban, you can also recieve emails or have an Output.txt created and updated with time and date of visits. It doesn't just have to be spiders, you can watch, log or ban any useragent!

How to install
Simply import the product ban_spider, the mod is active by default but none of the other options are turned on.

What is a UserAgent?
http://en.wikipedia.org/wiki/User_agent

Understanding a UserAgent string
http://user-agent-string.info/parse

Genuine User Getting Blocked?
https://vborg.vbsupport.ru/showpost....&postcount=105

Tools to help
http://whatsmyuseragent.com/SwitchingUserAgents.asp
http://www.botsvsbrowsers.com/SimulateUserAgent.asp

FAQ
https://vborg.vbsupport.ru/showpost....&postcount=137

How does it work?
https://vborg.vbsupport.ru/showpost....&postcount=381

What's a bot?
http://en.wikipedia.org/wiki/Spambot

How do i ban a bot?
https://vborg.vbsupport.ru/showpost....&postcount=318
https://vborg.vbsupport.ru/showpost....7&postcount=51

Where's output.txt located?
https://vborg.vbsupport.ru/showpost....&postcount=216

Bad bot lists
https://vborg.vbsupport.ru/showpost....&postcount=259
https://vborg.vbsupport.ru/showpost....&postcount=224
https://vborg.vbsupport.ru/showpost....&postcount=281

Tested on vb3.7.x, vB3.8.x , vB4.x.x but should work on any version.

__________________________________________________ __________________
Special thanks to:
Lior
KH99
BoP5
for helping me sort out a few issues

...and beta testers
ForceHSS (Special thanks to Force for latest testing)
ozzy47
GreyHost

If you use this please mark as INSTALLED

History
9th June 2011 Orginal xml added
12th June 2011 Added both email notification and text file logging
22nd June 2011 Version 2.0.0, Added create thread on activity

1. Added match facility you can now use something like Yandex and it will match MOZILLA/5.0 (COMPATIBLE; YANDEXBOT/3.0; +HTTP://YANDEX.COM/BOTS)
2. Added clickable link to visited thread

22nd September 2011 added user redirect url selection
08th October Beta testing started for thread creation.
20th October Beta testing started for emailing.
21st October Beta testing complete Ver 3.0.0 uploaded
29th October minor fix added to cope with empty userid on thread creation
30th October Beta testing automatic redirection to spiders/bots IP
31st October New xml uploaded with automatic redirect to IP
25th November Minor fix for blank forumid fixed
26th November 2011 Fixed version check & create thread Off by default
17th December 2014 Version 3.1.0 uploaded, Hook changed extra logging and statistics added by Ozzy47 (Chris)
18th December 2014 Version 3.1.1 uploaded, prevented spiders being counted when mod turned off.
17th December 2014 Version 3.1.2 uploaded, due to rogue code from another mod
The Bad Bots list is now included in the product
Please prune out all those that you wish to be able to see your site (i suggest you definately prune out "DA" and "Custo" :

Support will now only be given to those who have this mod marked as INSTALLED

[Simon Lloyd]

Originally Posted by ozzy47

I would not say a conflict, but perhaps a better hook to execute the mod. That is if the testing continues to provide the desired results.

Just bear in mind that any other hook you choose will need to be sufficient to perform the other tasks of the mod if you wished like sending the email or creating the threads. Some of the other runtime hooks will give errors or not work as expected especially with the thread creation, also keep in mind you need to redirect them before anything has loaded as it's this that is the basis of the mod - keeping resources for your members and not the bots

[CAG CheechDogg]

Originally Posted by Simon Lloyd

I agree in part, when you didnt have the block they were calling on every resource...php, mysql, cpu and ram, with the block they pretty much are just using ram as cpu and php time and response is minmal and as you are not loading anything else the ram isn't being maxed either. If you have whole country blocks that doesn't take as much checking as full octet ips like 192.161.0.1, if you have plenty of those then they are checked against each request, if you are blocking just 192.161 then its just one check against each request.

Im probably not explaining myself too well (it reads much better in my head ).

No my Man, you actually are explaining yourself very well lol ...

All I know is that I have not had any negative effects from doing the blocks and I also have just a list of single IPs ..... and let me tell you , that list is long as hell lol ....

[princesspepper]

Originally Posted by Max Taxable

Just don't redirect to any of your own pages - feedback loop danger.

EDIT TO ADD: I was right about the hook conflict with some other mod(s) Ozzy?

Thanks, but I'm still unsure what the benefit would be to redirect back to the source. Would it make them aware you don't want them sooner?

[Simon Lloyd]

Originally Posted by princesspepper

Thanks, but I'm still unsure what the benefit would be to redirect back to the source. Would it make them aware you don't want them sooner?

It really doesn't matter, they are redirected with a 301 which is a permanent redirect, so they will always see the url they tried to crawl as the one you send them to. Like i said, i coded that in to send them back to themselves so they have less resources to be crawling other peoples sites - it's only fair!

[Alan_SP]

Originally Posted by CAG CheechDogg

the best thing to do for me was also add a huge IP block to my htaccess file that completely blocks all of China and a couple other Asian countries from accessing my site ...

Would you share you CIDR list? Not in this thread, but maybe make a new thread?

[princesspepper]

Originally Posted by Simon Lloyd

i coded that in to send them back to themselves so they have less resources to be crawling other peoples sites - it's only fair!

Thanks, that is all I wanted to know.

[Simon Lloyd]

Princesspepper could you mark this as installed please

[CAG CheechDogg]

Originally Posted by Alan_SP

Would you share you CIDR list? Not in this thread, but maybe make a new thread?

You can get it here my Man : https://vborg.vbsupport.ru/showthrea...134.184.0%2F21

[Gadget_Guy]

Hey Ozzy,

Are we any closer to an alternative or modification to this so that we can get better blocking in place?

I am willing to test on my site as I am still getting hit hard by spiders even with the mod in place.

D.

[Simon Lloyd]

Hey Gadget Guy, no disrespect but the mod here is mine and isn't marked as reusable code, Ozzy may post what he's tried or done but wont necessarily be added to this mod, however Ozzy has developed one like this with other measures, you can get it at his site.

If you are being hit by spiders with this mod in place it will be because there is an anomaly in your list, this list isn't exhaustive but here's a few reasons why:
Entry in list has a leading or trailing space
Entry has a typo of some sort
Entry doesn't actually represent the bot you think it does (i.e Ahrefsbot I believe has a different name in the UA)
Mod run order may conflict with another mod using the same hook
There are other reasons but those should get you going!