Miscellaneous Hacks - Ban Spiders by User Agent

What this mod does
With this mod you can enter User Agents to watch or ban, you can also recieve emails or have an Output.txt created and updated with time and date of visits. It doesn't just have to be spiders, you can watch, log or ban any useragent!

How to install
Simply import the product ban_spider, the mod is active by default but none of the other options are turned on.

What is a UserAgent?
http://en.wikipedia.org/wiki/User_agent

Understanding a UserAgent string
http://user-agent-string.info/parse

Genuine User Getting Blocked?
https://vborg.vbsupport.ru/showpost....&postcount=105

Tools to help
http://whatsmyuseragent.com/SwitchingUserAgents.asp
http://www.botsvsbrowsers.com/SimulateUserAgent.asp

FAQ
https://vborg.vbsupport.ru/showpost....&postcount=137

What's a bot?
http://en.wikipedia.org/wiki/Spambot

How do i ban a bot?
https://vborg.vbsupport.ru/showpost....&postcount=318
https://vborg.vbsupport.ru/showpost....7&postcount=51

Where's output.txt located?
https://vborg.vbsupport.ru/showpost....&postcount=216

Bad bot lists
https://vborg.vbsupport.ru/showpost....&postcount=259
https://vborg.vbsupport.ru/showpost....&postcount=224
https://vborg.vbsupport.ru/showpost....&postcount=281

VB4.x Version of Ban Spiders By User Agent

Tested on vb3.7.x, vB3.8.x but should work on any version.

__________________________________________________ __________________
Special thanks to:
Lior
KH99
BoP5
for helping me sort out a few issues

...and beta testers
ForceHSS (Special thanks to Force for latest testing)
ozzy47
GreyHost

If you use this please mark as INSTALLED

History
9th June 2011 Orginal xml added
12th June 2011 Added both email notification and text file logging
22nd June 2011 Version 2.0.0, Added create thread on activity

1. Added match facility you can now use something like Yandex and it will match MOZILLA/5.0 (COMPATIBLE; YANDEXBOT/3.0; +HTTP://YANDEX.COM/BOTS)
2. Added clickable link to visited thread

22nd September 2011 added user redirect url selection
08th October Beta testing started for thread creation.
20th October Beta testing started for emailing.
21st October Beta testing complete Ver 3.0.0 uploaded
29th October minor fix added to cope with empty userid on thread creation
30th October Beta testing automatic redirection to spiders/bots IP
31st October New xml uploaded with automatic redirect to IP
25th November Minor fix for blank forumid fixed
26th November 2011 Fixed version check & create thread Off by default
17th December 2014 Version 3.1.0 uploaded, Extra logging and statistics added by Ozzy47 (Chris)
18th December 2014 Version 3.1.2 uploaded, due to rogue process from other mod
18th December 2014 Version 3.1.3 uploaded, due to previous one being VB4 mistakingly uploaded

The Bad Bots list is now included in the product
Please prune out all those that you wish to be able to see your site (i suggest you definately prune out "DA" and "Custo" :

Support will now only be given to those who have this mod marked as INSTALLED

[Simon Lloyd]

As you've downloaded thsi could you mark it as installed?
This particular mod is for banning User Agents, i do have another that's specifically for IP's https://vborg.vbsupport.ru/showthread.php?t=268146

Keep an eye out for my upcoming geo-banning of users

[Midohash]

Originally Posted by Simon Lloyd

Keep an eye out for my upcoming geo-banning of users

That will be another great mod Simon :up:, can I add this phrase to the list of user agents: Baiduspider/2.0 or just Baidu or Baiduspider?

[Simon Lloyd]

simply Baidu will do it

I'm curently working on geo-banning and geo-targeting for ads (so you can maximise your revenue).

[Midohash]

Originally Posted by Simon Lloyd

simply Baidu will do it

I'm curently working on geo-banning and geo-targeting for ads (so you can maximise your revenue).

Superb :up: ... does that mean I can allow traffic from some countries to google adsense ads and block traffic to ads from other countries with low click revenue?

[Simon Lloyd]

When i release it you will be able to target one country with ad_navbar_below and ad_ foorter everyone else will see what they always have seen!, thats the lite version, the full version will cater for multiple countries where you can designate ads to be shown in any current vbulletin ad location

[lebowski99]

Simon, I have a dumb question for you. I installed your mod but when I go into the "Who's Online" page I can still see bots hammering away at restricted areas in my site. How do I go about adding the user agent to the spider list on your add-on control panel? I figured out how to display the full user agent string. For example, I have one right now that reports:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts; .NET CLR 1.1.4322; PeoplePal

Looking at this string, what do I enter into the Spider List in the Ban Spider settings?

[Simon Lloyd]

Just enter funwebproducts
You can use the entire string if you wish or even peoplepal, you will see the bots who are in the banned list for anything up to an hour after install and you may on the odd occassion see them in the list but rest assured that they are being redirected, if you notice them there all the time still then pm me admin access to your site with permissions and i'll sort that for you

See the pic for adding to the list

Attachment 136597

[lebowski99]

Originally Posted by Simon Lloyd

Just enter funwebproducts
You can use the entire string if you wish or even peoplepal, you will see the bots who are in the banned list for anything up to an hour after install and you may on the odd occassion see them in the list but rest assured that they are being redirected, if you notice them there all the time still then pm me admin access to your site with permissions and i'll sort that for you

See the pic for adding to the list

Attachment 136597

Sounds easy enough. But here is one from russia and the user agent is listed simply as:

Opera/9.00 (Windows NT 5.1; U; en)

What do I enter for this one? Seems like if I enter the entire string, it would potentially block a legit user.

[Simon Lloyd]

Nope enter the entire string!, if by some very remote chance you find you've blocked a legit user then let me know and we'll work something out.

[trotskid]

Installed, thanks!