Miscellaneous Hacks - Ban Spiders by User Agent

What this mod does
With this mod you can enter User Agents to watch or ban, you can also recieve emails or have an Output.txt created and updated with time and date of visits. It doesn't just have to be spiders, you can watch, log or ban any useragent!

How to install
Simply import the product ban_spider, the mod is active by default but none of the other options are turned on.

What is a UserAgent?
http://en.wikipedia.org/wiki/User_agent

Understanding a UserAgent string
http://user-agent-string.info/parse

Genuine User Getting Blocked?
https://vborg.vbsupport.ru/showpost....&postcount=105

Tools to help
http://whatsmyuseragent.com/SwitchingUserAgents.asp
http://www.botsvsbrowsers.com/SimulateUserAgent.asp

FAQ
https://vborg.vbsupport.ru/showpost....&postcount=137

How does it work?
https://vborg.vbsupport.ru/showpost....&postcount=381

What's a bot?
http://en.wikipedia.org/wiki/Spambot

How do i ban a bot?
https://vborg.vbsupport.ru/showpost....&postcount=318
https://vborg.vbsupport.ru/showpost....7&postcount=51

Where's output.txt located?
https://vborg.vbsupport.ru/showpost....&postcount=216

Bad bot lists
https://vborg.vbsupport.ru/showpost....&postcount=259
https://vborg.vbsupport.ru/showpost....&postcount=224
https://vborg.vbsupport.ru/showpost....&postcount=281

Tested on vb3.7.x, vB3.8.x , vB4.x.x but should work on any version.

__________________________________________________ __________________
Special thanks to:
Lior
KH99
BoP5
for helping me sort out a few issues

...and beta testers
ForceHSS (Special thanks to Force for latest testing)
ozzy47
GreyHost

If you use this please mark as INSTALLED

History
9th June 2011 Orginal xml added
12th June 2011 Added both email notification and text file logging
22nd June 2011 Version 2.0.0, Added create thread on activity

1. Added match facility you can now use something like Yandex and it will match MOZILLA/5.0 (COMPATIBLE; YANDEXBOT/3.0; +HTTP://YANDEX.COM/BOTS)
2. Added clickable link to visited thread

22nd September 2011 added user redirect url selection
08th October Beta testing started for thread creation.
20th October Beta testing started for emailing.
21st October Beta testing complete Ver 3.0.0 uploaded
29th October minor fix added to cope with empty userid on thread creation
30th October Beta testing automatic redirection to spiders/bots IP
31st October New xml uploaded with automatic redirect to IP
25th November Minor fix for blank forumid fixed
26th November 2011 Fixed version check & create thread Off by default
17th December 2014 Version 3.1.0 uploaded, Hook changed extra logging and statistics added by Ozzy47 (Chris)
18th December 2014 Version 3.1.1 uploaded, prevented spiders being counted when mod turned off.
17th December 2014 Version 3.1.2 uploaded, due to rogue code from another mod
The Bad Bots list is now included in the product
Please prune out all those that you wish to be able to see your site (i suggest you definately prune out "DA" and "Custo" :

Support will now only be given to those who have this mod marked as INSTALLED

[Simon Lloyd]

Please mark it as installed

[Simon Lloyd]

Are you dead sure on those early IE's?

Max, could you mark as installed please?

[Max Taxable]

Originally Posted by Simon Lloyd

Are you dead sure on those early IE's?

I am dead sure the percentage of human beings still using these dinosaurs is infinitesimally small, so small they're not worth worrying about losing. (None of my 3,200+ users have these, for example)

I am also dead sure that entering these into your Mod doesn't interfere with IE 7,8,9 etc. Tested and verified.

I am also dead sure that the IsBot Mod I have is still working, but that since I put the dinosaur IE's in your Mod - it went from catching 40-50 bot registration attempts per day to catching only one or two!

The early IE's are 99.9% of the spam bot problem on the web, because these are easily infected to become botnet zombies. Human spammers are extremely rare, because think about it - if you have to pay someone to spam it kind of defeats the purpose of spamming.

I used to get 1,500 or so visits a day from these early IE computers, and spent months analyzing them and their origins. Never found one that looked like a Human. It is the 21st Century already, and I think it is high time webmasters not only stopped supporting early IE, but should also take steps to just plain block them. If the FBI and Microsoft really wanted to stop the botnet problem, MS would revoke the registration of these, or automatically upgrade them.

I used to use a script that did just that - would detect early IE and install the latest version of firefox, making it the default browser on that computer - using the same exploits that made them botnet zombies in the first place. I virtually wiped out a entire botnet that way, back in 2006 while one of my sites was undergoing a DDoS attack from one.

Your Mod is by far the best weapon against the botnets yet, and I have been studying them and fighting them for at least 10 years.

Max, could you mark as installed please?

I did, on the 3.8.x version I run.

[Simon Lloyd]

, thanks and thanks!

[manning]

Originally Posted by Wayne Luke

I banned them at the server level. Not catering to the Chinese or Asian market and never will cater to the Chinese or Asian market so don't need them to index my site.

Interesting idea - my forum really doesnt cater to Asian markets either no Russian or pretty much any place other than USA maybe UK ... What if I add ALLOW for those IPS and deny for everyone else... that makes htaccess huge - what affect will that have on load time? Course if they use a proxy in one of the other locations theyd still get in..... damn idiots!

[BadgerDog]

Just for my clarity ...

I still get spiders appearing in PaulM's guest list and I understand from previous posts why. I also still see spiders active in my "Who's On-line" listing, but I understand that doesn't mean they actually are on the site, but have showed and been redirected?

As a test, I turned ON for a few minutes the post in thread option, captured a few posts and then turned it OFF.

Here's a typical thread it started:

Activity from Bot No. 7 (Baiduspider) in your list

Date and Time: 03-16-2012 06:57:28
Associated Username (if any): Unregistered
Matched bots[7]: Baiduspider
With User Agent: MOZILLA/5.0 (COMPATIBLE; BAIDUSPIDER/2.0; +HTTP://WWW.BAIDU.COM/SEARCH/SPIDER.HTML)

Does this mean that in fact that the Baidu spider has been caught by this mod and redirected elsewhere? Does it mean that the mod is actually working, in spite of what appears in the "Who's On-line" listing?

Thanks ..

Regards,
Doug

[Max Taxable]

BadgerDog that's strange, I never see any of the banned user agents either in who's online or in Paul's Track Guest Visits Mod.

[Simon Lloyd]

Originally Posted by BadgerDog

Just for my clarity ...

I still get spiders appearing in PaulM's guest list and I understand from previous posts why. I also still see spiders active in my "Who's On-line" listing, but I understand that doesn't mean they actually are on the site, but have showed and been redirected?

As a test, I turned ON for a few minutes the post in thread option, captured a few posts and then turned it OFF.

Here's a typical thread it started:



Does this mean that in fact that the Baidu spider has been caught by this mod and redirected elsewhere? Does it mean that the mod is actually working, in spite of what appears in the "Who's On-line" listing?

Thanks ..

Regards,
Doug

It may be that the mod is conflicting with some other mod, if you want to pm me admin access with permissions i'll take a look for you

[baileyjojoms]

Just a hint to anyone with Baidu Spider issues. This Mod works great, but after getting 30,000 spider bans I had enough. I contact Baidu via their Spider Complaint section on their webpage, and they have halted crawling my site. This request was processed within 3 working days. I haven't seen a hint of Baidu since then.

[Simon Lloyd]

Thats great news, i cnat believe you actually logged all those denials , great info anyway as Baidu doesn't follow robots.txt (which they claim it does).