Miscellaneous Hacks - Ban Spiders by User Agent

What this mod does
With this mod you can enter User Agents to watch or ban, you can also recieve emails or have an Output.txt created and updated with time and date of visits. It doesn't just have to be spiders, you can watch, log or ban any useragent!

How to install
Simply import the product ban_spider, the mod is active by default but none of the other options are turned on.

What is a UserAgent?
http://en.wikipedia.org/wiki/User_agent

Understanding a UserAgent string
http://user-agent-string.info/parse

Genuine User Getting Blocked?
https://vborg.vbsupport.ru/showpost....&postcount=105

Tools to help
http://whatsmyuseragent.com/SwitchingUserAgents.asp
http://www.botsvsbrowsers.com/SimulateUserAgent.asp

FAQ
https://vborg.vbsupport.ru/showpost....&postcount=137

How does it work?
https://vborg.vbsupport.ru/showpost....&postcount=381

What's a bot?
http://en.wikipedia.org/wiki/Spambot

How do i ban a bot?
https://vborg.vbsupport.ru/showpost....&postcount=318
https://vborg.vbsupport.ru/showpost....7&postcount=51

Where's output.txt located?
https://vborg.vbsupport.ru/showpost....&postcount=216

Bad bot lists
https://vborg.vbsupport.ru/showpost....&postcount=259
https://vborg.vbsupport.ru/showpost....&postcount=224
https://vborg.vbsupport.ru/showpost....&postcount=281

Tested on vb3.7.x, vB3.8.x , vB4.x.x but should work on any version.

__________________________________________________ __________________
Special thanks to:
Lior
KH99
BoP5
for helping me sort out a few issues

...and beta testers
ForceHSS (Special thanks to Force for latest testing)
ozzy47
GreyHost

If you use this please mark as INSTALLED

History
9th June 2011 Orginal xml added
12th June 2011 Added both email notification and text file logging
22nd June 2011 Version 2.0.0, Added create thread on activity

1. Added match facility you can now use something like Yandex and it will match MOZILLA/5.0 (COMPATIBLE; YANDEXBOT/3.0; +HTTP://YANDEX.COM/BOTS)
2. Added clickable link to visited thread

22nd September 2011 added user redirect url selection
08th October Beta testing started for thread creation.
20th October Beta testing started for emailing.
21st October Beta testing complete Ver 3.0.0 uploaded
29th October minor fix added to cope with empty userid on thread creation
30th October Beta testing automatic redirection to spiders/bots IP
31st October New xml uploaded with automatic redirect to IP
25th November Minor fix for blank forumid fixed
26th November 2011 Fixed version check & create thread Off by default
17th December 2014 Version 3.1.0 uploaded, Hook changed extra logging and statistics added by Ozzy47 (Chris)
18th December 2014 Version 3.1.1 uploaded, prevented spiders being counted when mod turned off.
17th December 2014 Version 3.1.2 uploaded, due to rogue code from another mod
The Bad Bots list is now included in the product
Please prune out all those that you wish to be able to see your site (i suggest you definately prune out "DA" and "Custo" :

Support will now only be given to those who have this mod marked as INSTALLED

[Simon Lloyd]

Well i suppose the secondd of those two links has more explanation but i didn't get that from your post, i thought you wanted links to lists posting, but for my money that can be a bad thing, people would then blindly enter all the bots to the banning list when really, dependant on their content, they will want some of those scraping or visiting their site, it's a personal preference really.

I built this because the bots were killing my bandwidth and making the forum slow so i ban the more agressive ones like Baidu, why on earth would they need 215 bots indexing my site?, so ban them in favour for the less aggressive

Anyway as an information post this link explains what they are http://en.wikipedia.org/wiki/Spambot

[Max Taxable]

Originally Posted by Simon Lloyd

i ban the more agressive ones like Baidu, why on earth would they need 215 bots indexing my site?

Yeah, 215 Chinese bots leeching resources, to allegedly index your site for people who most likely can never see it. Most US sites are blocked in China. Baidu makes no sense at all, it certainly doesn't help anybody. By far the worst behaving bot out there, it totally ignores robots.txt.

[ForceHSS]

Originally Posted by Boofo

A link to those lists would be a good addition, also.

https://vborg.vbsupport.ru/showpost....&postcount=224

[Simon Lloyd]

Lol, thanks Force

[spillage]

Since upgrading to vB4.1.10, I've noticed (occasionally) some spiders in the list still showing up online.
Anyone else having this issue, and any ideas what's going on?

TiA

[Simon Lloyd]

You shouldn't unless they are using a different UA, the spiders xml that you can download from Mosh's site (vb.com thread) may have them in the list with the same name but their UA may be different to that in the list

[spillage]

A couple of examples of ones in my list that (recently) show as online;
AhrefsBot
Exabot
gigabaz
Heritrix
Majestics MJ12bot

I though this hack picked up on all things similar (name based)?... ie "Majestics" should also take out "Majestics MJ12bot", regardless of the software they're using to access the site.

How do we include same name spiders in our list that are using different UA's?

[Simon Lloyd]

You are right entering Exabot should prevent that bot viewing your site, however entering "Majestics MJ12bot" will not stop a bot with either "Majestics" or "MJ12bot" it will only stop bots with that entire phrase as part of their UA, so if you want to ban that bot enter the seperated values.

Can you give me a link to your site?

[spillage]

Simon, I thought about that and included the individual entries when the issue first arose.

nscale.net ... however, the WGO block is only visible to members... PM me if you want temporary access.

[TombstoneWarrior]

what does this option mean? " Create new thread for each UA detection (be aware that this can cause hundreds of threads at first until spiders get the mesage!)
You can use this even if you aren't using the ban option above" also what does this option do. will it creat a thread in my forum?

ALSO WHY IN A DISCRIPTION FROMA MEMBER DO THEY HAVE THIS OPTION SET TO YES??