Miscellaneous Hacks - Ban Spiders by User Agent

What this mod does
With this mod you can enter User Agents to watch or ban, you can also recieve emails or have an Output.txt created and updated with time and date of visits. It doesn't just have to be spiders, you can watch, log or ban any useragent!

How to install
Simply import the product ban_spider, the mod is active by default but none of the other options are turned on.

What is a UserAgent?
http://en.wikipedia.org/wiki/User_agent

Understanding a UserAgent string
http://user-agent-string.info/parse

Genuine User Getting Blocked?
https://vborg.vbsupport.ru/showpost....&postcount=105

Tools to help
http://whatsmyuseragent.com/SwitchingUserAgents.asp
http://www.botsvsbrowsers.com/SimulateUserAgent.asp

FAQ
https://vborg.vbsupport.ru/showpost....&postcount=137

What's a bot?
http://en.wikipedia.org/wiki/Spambot

How do i ban a bot?
https://vborg.vbsupport.ru/showpost....&postcount=318
https://vborg.vbsupport.ru/showpost....7&postcount=51

Where's output.txt located?
https://vborg.vbsupport.ru/showpost....&postcount=216

Bad bot lists
https://vborg.vbsupport.ru/showpost....&postcount=259
https://vborg.vbsupport.ru/showpost....&postcount=224
https://vborg.vbsupport.ru/showpost....&postcount=281

VB4.x Version of Ban Spiders By User Agent

Tested on vb3.7.x, vB3.8.x but should work on any version.

__________________________________________________ __________________
Special thanks to:
Lior
KH99
BoP5
for helping me sort out a few issues

...and beta testers
ForceHSS (Special thanks to Force for latest testing)
ozzy47
GreyHost

If you use this please mark as INSTALLED

History
9th June 2011 Orginal xml added
12th June 2011 Added both email notification and text file logging
22nd June 2011 Version 2.0.0, Added create thread on activity

1. Added match facility you can now use something like Yandex and it will match MOZILLA/5.0 (COMPATIBLE; YANDEXBOT/3.0; +HTTP://YANDEX.COM/BOTS)
2. Added clickable link to visited thread

22nd September 2011 added user redirect url selection
08th October Beta testing started for thread creation.
20th October Beta testing started for emailing.
21st October Beta testing complete Ver 3.0.0 uploaded
29th October minor fix added to cope with empty userid on thread creation
30th October Beta testing automatic redirection to spiders/bots IP
31st October New xml uploaded with automatic redirect to IP
25th November Minor fix for blank forumid fixed
26th November 2011 Fixed version check & create thread Off by default
17th December 2014 Version 3.1.0 uploaded, Extra logging and statistics added by Ozzy47 (Chris)
18th December 2014 Version 3.1.2 uploaded, due to rogue process from other mod
18th December 2014 Version 3.1.3 uploaded, due to previous one being VB4 mistakingly uploaded

The Bad Bots list is now included in the product
Please prune out all those that you wish to be able to see your site (i suggest you definately prune out "DA" and "Custo" :

Support will now only be given to those who have this mod marked as INSTALLED

[accludetuner]

Trying the ban IP mod now.

On a side note, I help several people run other VB sites and I've been installing this spider ban mod as a default thing on all of their sites and putting in my list of user agents to ban/block. I recently installed it on a 3.6 site and it doesn't seem to be working. Is there a different hook I need to use for it to work with 3.6? Thought you might know.

EDIT: Maybe it is working and I spoke too soon. I have that pesky Baidu spider banned due to the amount of indexing it does without any regard to server resources. There were about 50 Baidu bots crawling the site but the number seems to be gradually going down (at about 20 right now). I'll keep checking it to see if Baidu disappears for good and then I'll know for sure whether it's working on 3.6 or not.

[lgpaul]

It works! Thanks!

[pitzerwm]

Thank you for creating a great mod, and thanks to Max for turning me on to it.

A question: in Who is online, in the IP column, right now I have 2 guest where the IP is totally blank. I suppose that there is an explanation for this, but is there anything that you can do about it?

[Midohash]

Fantastic ... marked as Installed ... many thanks
Just a little query! ... If I need to block unwanted visitor or member, should I put his/her IP in the spider list? ... And what is the difference between denying the IP address on cpanel or .htaccess? ... Also can I block a proxy server through this mod?

[Simon Lloyd]

For IP's you should use my other mod https://vborg.vbsupport.ru/showthread.php?t=268146 as this mod processes the useragent string and not the IP address, if you deny in cpanel or .htaccess then the user of that IP will never get to see your site or files, my mod redirects them before your site loads properly. Also with cpanel and .htaccess you have to of course log in and edit them each time you want to ban or remove a ban, mines a little more user friendly

[Simon Lloyd]

Originally Posted by pitzerwm

Thank you for creating a great mod, and thanks to Max for turning me on to it.

A question: in Who is online, in the IP column, right now I have 2 guest where the IP is totally blank. I suppose that there is an explanation for this, but is there anything that you can do about it?

Sorry, again you need this https://vborg.vbsupport.ru/showthread.php?t=268146 for IP's, this deals with the user agent string not IP's, if their IP is blank then they are cloaking it somehow, if you choose to see the User Agent (via the dropdown at the bottom of the WHo's Online window you can copy their user agent string, pop it in the list to ban and then watch them disappear

[Midohash]

Originally Posted by Simon Lloyd

For IP's you should use my other mod https://vborg.vbsupport.ru/showthread.php?t=268146 as this mod processes the useragent string and not the IP address, if you deny in cpanel or .htaccess then the user of that IP will never get to see your site or files, my mod redirects them before your site loads properly. Also with cpanel and .htaccess you have to of course log in and edit them each time you want to ban or remove a ban, mines a little more user friendly

Thanks Simon :up:, regarding your other mod is it possible to be modified in the future to block users from anonymous proxy? ... That will be a great add on ... However I note a significant drop in the number of my site visitors after installing the mod, is that attributed to blocking bad bots only or could be also some normal visitors are not able now to access my forum? ... Also is the drop in the visitor numbers would affect my over all alexa rank? ... Regards,

[pitzerwm]

I installed this and with Ban IPs apparently have removed a large percentage of my issues.

I was wondering if I get this as the "guest" dhcp-0-24-b2-58-41-4a.cpe.eaglecable.net

Can I put this total "agent" in, and it will ban this person?

Thanks

[photonetau]

I installed this about 9 hours ago and baidu is still hitting at the rate of 200 hits an hour is that usual , how long before they give up?
BTW they are redirected to the default site.

[Simon Lloyd]

Originally Posted by Midohash

Thanks Simon :up:, regarding your other mod is it possible to be modified in the future to block users from anonymous proxy? ... That will be a great add on ... However I note a significant drop in the number of my site visitors after installing the mod, is that attributed to blocking bad bots only or could be also some normal visitors are not able now to access my forum? ... Also is the drop in the visitor numbers would affect my over all alexa rank? ... Regards,

It could be modified to ban those without IP and this one could be modified to ban those without user agent but would need a lot of testing!

Yes, your visitor drop is due tothe banning, you'll find (especially if you installed the spiders on forumhome mod) that most of the visitors were bots in the list like Baiduspider, they're insatiable. Bots don't make up part of your alexa ranking, believe it or not Alexa makes it's assumptions from the amount of visitors to your site that have the Alexa toolbar installed

Originally Posted by pitzerwm

I installed this and with Ban IPs apparently have removed a large percentage of my issues.

I was wondering if I get this as the "guest" dhcp-0-24-b2-58-41-4a.cpe.eaglecable.net

Can I put this total "agent" in, and it will ban this person?

Thanks

Yes of course, the exact agent string will help zero in on that person/bot...etc, i added partial matching because of bots changing their UAs..etc but this mod was always built on using the entire string

Originally Posted by photonetau

I installed this about 9 hours ago and baidu is still hitting at the rate of 200 hits an hour is that usual , how long before they give up?
BTW they are redirected to the default site.

I guess you are still seeing Baidu because you are using Paul M's who visited mod, both his mod and mine are doing their job, his picks them up because they make a direct call to a thread or post...etc and get logged but as soon as they attempt to access the thread or post they are redirected to whichever way you choose with a 301 redirect header

If you are seeing Baiduspider and not using Paul's mod then pm me your site url and access details and we'll get you sorted!