The Arcive of Official vBulletin Modifications Site.

Ban Spiders by User Agent · **Released**: 08-08-2011

What this mod does
With this mod you can enter User Agents to watch or ban, you can also recieve emails or have an Output.txt created and updated with time and date of visits. It doesn't just have to be spiders, you can watch, log or ban any useragent!

How to install
Simply import the product ban_spider, the mod is active by default but none of the other options are turned on.

What is a UserAgent?
http://en.wikipedia.org/wiki/User_agent

Understanding a UserAgent string
http://user-agent-string.info/parse

Genuine User Getting Blocked?
https://vborg.vbsupport.ru/showpost....&postcount=105

Tools to help
http://whatsmyuseragent.com/SwitchingUserAgents.asp
http://www.botsvsbrowsers.com/SimulateUserAgent.asp

FAQ
https://vborg.vbsupport.ru/showpost....&postcount=137

How does it work?
https://vborg.vbsupport.ru/showpost....&postcount=381

What's a bot?
http://en.wikipedia.org/wiki/Spambot

How do i ban a bot?
https://vborg.vbsupport.ru/showpost....&postcount=318
https://vborg.vbsupport.ru/showpost....7&postcount=51

Where's output.txt located?
https://vborg.vbsupport.ru/showpost....&postcount=216

Bad bot lists
https://vborg.vbsupport.ru/showpost....&postcount=259
https://vborg.vbsupport.ru/showpost....&postcount=224
https://vborg.vbsupport.ru/showpost....&postcount=281

Tested on vb3.7.x, vB3.8.x , vB4.x.x but should work on any version.

__________________________________________________ __________________
Special thanks to:
Lior
KH99
BoP5
for helping me sort out a few issues

...and beta testers
ForceHSS (Special thanks to Force for latest testing)
ozzy47
GreyHost

If you use this please mark as INSTALLED

History
9th June 2011 Orginal xml added
12th June 2011 Added both email notification and text file logging
22nd June 2011 Version 2.0.0, Added create thread on activity

Added match facility you can now use something like Yandex and it will match MOZILLA/5.0 (COMPATIBLE; YANDEXBOT/3.0; +HTTP://YANDEX.COM/BOTS)
Added clickable link to visited thread

22nd September 2011 added user redirect url selection
08th October Beta testing started for thread creation.
20th October Beta testing started for emailing.
21st October Beta testing complete Ver 3.0.0 uploaded
29th October minor fix added to cope with empty userid on thread creation
30th October Beta testing automatic redirection to spiders/bots IP
31st October New xml uploaded with automatic redirect to IP
25th November Minor fix for blank forumid fixed
26th November 2011 Fixed version check & create thread Off by default
17th December 2014 Version 3.1.0 uploaded, Hook changed extra logging and statistics added by Ozzy47 (Chris)
18th December 2014 Version 3.1.1 uploaded, prevented spiders being counted when mod turned off.
17th December 2014 Version 3.1.2 uploaded, due to rogue code from another mod
The Bad Bots list is now included in the product
Please prune out all those that you wish to be able to see your site (i suggest you definately prune out "DA" and "Custo" :

Support will now only be given to those who have this mod marked as INSTALLED

Simon Lloyd · #72 09-19-2011, 07:05 AM

Your second UA can access your site (i just used the UA simulator from the link i posted above)

Simon Lloyd · #73 09-19-2011, 07:14 AM

post just to get to page 5 as this thread appears to be faulty!

niteflyer32 · #74 09-19-2011, 07:44 AM

Thanks for the help on the UA list.

When I tested the 2nd UA I listed above with your UA test website http://www.botsvsbrowsers.com/SimulateUserAgent.asp I see our forum but I also see a "Request Status: 500 : Internal Server Error" at the top. Is that caused by our US block list or is that a server setting issue I need to talk to our webhost about? I also get a "500 : Internal Server Error" when trying to verify we have our Comscore analytic code (like Google Analytics) inserted on the forum.

What is the status of the spiders Majestics MJ12bot Spider, Speedy Spider and Voila Spider? I've searched them and get info that is dated or conflicting on if they are good or bad. They are hitting our site pretty often.

The page 5 weirdness of this thread you're seeing appears to be okay on my end.

BadgerDog · #75 09-19-2011, 11:51 AM

Installed with thanks ..

Unfortunately, I'm still getting Baidu spiders, even 4 days after installing this.

Attached is screen pic. What am I doing wrong?

Regards,
Doug

Simon Lloyd · #76 09-19-2011, 12:59 PM

For now there is an issue when banning bots and having one of the notification enabled (either create thread or Output.txt file on server), so if you have those enabled please disable them, to ban bots for now you must only have the mod activated and ban bots in list selected (and of course bots that you want to ban, do that and all will be good

You will recieve notice when i solve the "working together" issue.

Simon Lloyd · #77 09-19-2011, 01:04 PM

@niteflyer32, try turning the mod off then trying the UA at the UA simulation site and see what it returns, the mod shouldn't cause a 500 error and then show you the site, if you don't have access you simply get redirected so you wouldn't see your site.

I haven't researched the spiders, i built this mod to cut down on my server load as Baidu were hammering it, i have between 200 and 350 Baidu at my site at any one time, and because the index so vigourously their demand on the server is huge (although while im working on the issues with the notification i am allowing all bots at the moment).

BadgerDog · #78 09-19-2011, 01:48 PM

Quote:

Originally Posted by Simon Lloyd

For now there is an issue when banning bots and having one of the notification enabled (either create thread or Output.txt file on server), so if you have those enabled please disable them, to ban bots for now you must only have the mod activated and ban bots in list selected (and of course bots that you want to ban, do that and all will be good

You will recieve notice when i solve the "working together" issue.

Thanks ..

I've turned OFF logging (email notifications were already off) and I'll monitor it now ...

Regards,
Doug

BadgerDog · #79 09-22-2011, 10:09 AM

Quote:

Originally Posted by BadgerDog

Thanks ..

I've turned OFF logging (email notifications were already off) and I'll monitor it now ...

Regards,
Doug

That doesn't work either....

Still getting lots of Baidu and Yandex spiders ...

I'm not sure this mod is working at all, regardless of any options set, or turned ON or OFF ...

Regards,
Doug

smirkley · #80 09-22-2011, 04:08 PM

Still testing but I can say so far,... NICE !!

Thank you.

I am only banning 4 useragnts at the moment, but I wish to ask is there a condensed version of 'must ban' useragents off that list here, as compared to the whole list? I dont want to go crazy and ban too much especially if it hurts my membership or adsense rev.

So far I ban:

Baidu
Yeti
Twiceler
Yandex

Simon Lloyd · #81 09-22-2011, 04:52 PM

Quote:

Originally Posted by BadgerDog

That doesn't work either....

Still getting lots of Baidu and Yandex spiders ...

I'm not sure this mod is working at all, regardless of any options set, or turned ON or OFF ...

Regards,
Doug

if you want to pm me admin access details and url i'll take a look

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.05166 seconds Memory Usage 2,361KB Queries Executed 27 (?)
More Information
Template Usage: (1)SHOWTHREAD (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (1)ad_showthread_beforeqr (3)bbcode_quote (1)footer (1)forumjump (1)forumrules (1)gobutton (1)header (1)headinclude (1)modsystem_post (1)navbar (4)navbar_link (120)option (1)pagenav (1)pagenav_curpage (4)pagenav_pagelink (2)pagenav_pagelinkrel (11)post_thanks_box (16)post_thanks_box_bit (11)post_thanks_button (1)post_thanks_javascript (1)post_thanks_navbar_search (1)post_thanks_postbit (11)post_thanks_postbit_info (10)postbit (2)postbit_attachment (11)postbit_onlinestatus (11)postbit_wrapper (1)spacer_close (1)spacer_open (1)tagbit_wrapper Phrase Groups Available: global inlinemod postbit posting reputationlevel showthread	Included Files: ./showthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/functions_bigthree.php ./includes/class_postbit.php ./includes/class_bbcode.php ./includes/functions_reputation.php ./includes/functions_post_thanks.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete showthread_start showthread_getinfo forumjump showthread_post_start showthread_query_postids showthread_query bbcode_fetch_tags bbcode_create showthread_postbit_create postbit_factory postbit_display_start post_thanks_function_post_thanks_off_start post_thanks_function_post_thanks_off_end post_thanks_function_fetch_thanks_start fetch_musername post_thanks_function_fetch_thanks_end post_thanks_function_thanked_already_start post_thanks_function_thanked_already_end post_thanks_function_fetch_thanks_bit_start post_thanks_function_show_thanks_date_start post_thanks_function_show_thanks_date_end post_thanks_function_fetch_thanks_bit_end post_thanks_function_fetch_post_thanks_template_start post_thanks_function_fetch_post_thanks_template_end postbit_imicons bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete postbit_attachment postbit_display_complete post_thanks_function_can_thank_this_post_start pagenav_page pagenav_complete tag_fetchbit_complete forumrules navbits navbits_complete showthread_complete
Messages:

16 благодарности(ей) от:
ASCONAN, BlueCheri, Disco_Dave, ForceHSS, Gn_Snake, GreyGhost, htscpl, Michael.A, midnz, obglobal.net, puertoblack2003, RichieBoy67, rootsxrocks, tanzeelniazi, too_cool_3, victorvu

The Arcive of Official vBulletin Modifications Site.

It is not a VB3 engine, just a parsed copy!