Miscellaneous Hacks - Ban Spiders by User Agent

What this mod does
With this mod you can enter User Agents to watch or ban, you can also recieve emails or have an Output.txt created and updated with time and date of visits. It doesn't just have to be spiders, you can watch, log or ban any useragent!

How to install
Simply import the product ban_spider, the mod is active by default but none of the other options are turned on.

What is a UserAgent?
http://en.wikipedia.org/wiki/User_agent

Understanding a UserAgent string
http://user-agent-string.info/parse

Genuine User Getting Blocked?
https://vborg.vbsupport.ru/showpost....&postcount=105

Tools to help
http://whatsmyuseragent.com/SwitchingUserAgents.asp
http://www.botsvsbrowsers.com/SimulateUserAgent.asp

FAQ
https://vborg.vbsupport.ru/showpost....&postcount=137

What's a bot?
http://en.wikipedia.org/wiki/Spambot

How do i ban a bot?
https://vborg.vbsupport.ru/showpost....&postcount=318
https://vborg.vbsupport.ru/showpost....7&postcount=51

Where's output.txt located?
https://vborg.vbsupport.ru/showpost....&postcount=216

Bad bot lists
https://vborg.vbsupport.ru/showpost....&postcount=259
https://vborg.vbsupport.ru/showpost....&postcount=224
https://vborg.vbsupport.ru/showpost....&postcount=281

VB4.x Version of Ban Spiders By User Agent

Tested on vb3.7.x, vB3.8.x but should work on any version.

__________________________________________________ __________________
Special thanks to:
Lior
KH99
BoP5
for helping me sort out a few issues

...and beta testers
ForceHSS (Special thanks to Force for latest testing)
ozzy47
GreyHost

If you use this please mark as INSTALLED

History
9th June 2011 Orginal xml added
12th June 2011 Added both email notification and text file logging
22nd June 2011 Version 2.0.0, Added create thread on activity

1. Added match facility you can now use something like Yandex and it will match MOZILLA/5.0 (COMPATIBLE; YANDEXBOT/3.0; +HTTP://YANDEX.COM/BOTS)
2. Added clickable link to visited thread

22nd September 2011 added user redirect url selection
08th October Beta testing started for thread creation.
20th October Beta testing started for emailing.
21st October Beta testing complete Ver 3.0.0 uploaded
29th October minor fix added to cope with empty userid on thread creation
30th October Beta testing automatic redirection to spiders/bots IP
31st October New xml uploaded with automatic redirect to IP
25th November Minor fix for blank forumid fixed
26th November 2011 Fixed version check & create thread Off by default
17th December 2014 Version 3.1.0 uploaded, Extra logging and statistics added by Ozzy47 (Chris)
18th December 2014 Version 3.1.2 uploaded, due to rogue process from other mod
18th December 2014 Version 3.1.3 uploaded, due to previous one being VB4 mistakingly uploaded

The Bad Bots list is now included in the product
Please prune out all those that you wish to be able to see your site (i suggest you definately prune out "DA" and "Custo" :

Support will now only be given to those who have this mod marked as INSTALLED

[Simon Lloyd]

is it still posting threads? i'm using the vb3.8 version my thread creation is turned off and mine isn't creating them - post back please.

[dapence]

This seems to work well! I installed, activated, and the list of 'visitors' dropped quite around 2/3.

Keep up the good work. I think the people at vBulletin proper should get you on board and have this added as a default feature of vBulletin.

Help this guy keep the agent list up-to-date.

[Boofo]

Originally Posted by Simon Lloyd

is it still posting threads? i'm using the vb3.8 version my thread creation is turned off and mine isn't creating them - post back please.

Why not wrap the whole thread creation code in a setting conditional? That way it would only run if the setting was enabled.

[Simon Lloyd]

Boofo, after my recent update it now will not fire if there isn't a forumid present, if a username is left out it automatically enters the userid 1 (as most of us start our forums off with our username) and finally the create thread option which really should have been off by default (my bad!) is now set to off by default

[Simon Lloyd]

Originally Posted by dapence

This seems to work well! I installed, activated, and the list of 'visitors' dropped quite around 2/3.

Keep up the good work. I think the people at vBulletin proper should get you on board and have this added as a default feature of vBulletin.

Help this guy keep the agent list up-to-date.

Glad you like it , i doubt the guys at IB have this on their radar!

Can you mark this mod as installed please?

[Boofo]

Originally Posted by Simon Lloyd

Boofo, after my recent update it now will not fire if there isn't a forumid present, if a username is left out it automatically enters the userid 1 (as most of us start our forums off with our username) and finally the create thread option which really should have been off by default (my bad!) is now set to off by default

I saw that, but wrapping the code would have prevented the problem in the first place instead of having to add exit, was my point.

[Lee G]

Originally Posted by dapence

This seems to work well! I installed, activated, and the list of 'visitors' dropped quite around 2/3.

Keep up the good work. I think the people at vBulletin proper should get you on board and have this added as a default feature of vBulletin.

Help this guy keep the agent list up-to-date.

Loads of agents to consider if you look over at vbulletin
https://www.vbulletin.com/forum/show...d-spammer-trap
A good nine pages there

[Lee G]

Heres another for your low life with limited braincell capacity set forth and multiply deployment system

Firefox/3.0b5

Thats a free browser, beta version
Released: April 2, 2008

Had someone with a Romanian set of ips hit me with that

[accludetuner]

I just wanted to bump this thread and point out that this is by far one of the best and most effective anti-spam, anti-scraper, anti-harvester, anti-banned member, anti-WHATEVER mods available. It might take a little bit of time to get it setup for your specific forum, and for who you want/don't want to access your site, but once you have it setup, it's BULLETPROOF and very effective. If you're hoping to install a magical mod that magically gets rid of all the unwanted visitors from ever going to your site, keep living in magical fairy land because any effective solution is going to require some trial and error and some configuration. This mod is no exception, but once it's setup, it's absolutely fantastic!

I rated it 5 stars!

[Simon Lloyd]

Glad you like it, if you team it up with my Ban IP mod you'll get that better flexibility.