vb.org Archive - Miscellaneous Hacks - Ban Spiders by User Agent

vb.org Archive (https://vborg.vbsupport.ru/index.php)

- vBulletin 4.x Add-ons (https://vborg.vbsupport.ru/forumdisplay.php?f=245)

- - Miscellaneous Hacks - Ban Spiders by User Agent (https://vborg.vbsupport.ru/showthread.php?t=268208)

Ban Spiders by User Agent

What this mod does
With this mod you can enter User Agents to watch or ban, you can also recieve emails or have an Output.txt created and updated with time and date of visits. It doesn't just have to be spiders, you can watch, log or ban any useragent!

How to install
Simply import the product ban_spider, the mod is active by default but none of the other options are turned on.

What is a UserAgent?
http://en.wikipedia.org/wiki/User_agent

Understanding a UserAgent string
http://user-agent-string.info/parse

Genuine User Getting Blocked?
https://vborg.vbsupport.ru/showpost....&postcount=105

Tools to help
http://whatsmyuseragent.com/SwitchingUserAgents.asp
http://www.botsvsbrowsers.com/SimulateUserAgent.asp

FAQ
https://vborg.vbsupport.ru/showpost....&postcount=137

How does it work?
https://vborg.vbsupport.ru/showpost....&postcount=381

What's a bot?
http://en.wikipedia.org/wiki/Spambot

How do i ban a bot?
https://vborg.vbsupport.ru/showpost....&postcount=318
https://vborg.vbsupport.ru/showpost....7&postcount=51

Where's output.txt located?
https://vborg.vbsupport.ru/showpost....&postcount=216

Bad bot lists
https://vborg.vbsupport.ru/showpost....&postcount=259
https://vborg.vbsupport.ru/showpost....&postcount=224
https://vborg.vbsupport.ru/showpost....&postcount=281

Tested on vb3.7.x, vB3.8.x , vB4.x.x but should work on any version.

__________________________________________________ __________________
Special thanks to:
Lior
KH99
BoP5
for helping me sort out a few issues

...and beta testers
ForceHSS (Special thanks to Force for latest testing)
ozzy47
GreyHost

If you use this please mark as INSTALLED

History
9th June 2011 Orginal xml added
12th June 2011 Added both email notification and text file logging
22nd June 2011 Version 2.0.0, Added create thread on activity

Added match facility you can now use something like Yandex and it will match MOZILLA/5.0 (COMPATIBLE; YANDEXBOT/3.0; +HTTP://YANDEX.COM/BOTS)
Added clickable link to visited thread

22nd September 2011 added user redirect url selection
08th October Beta testing started for thread creation.
20th October Beta testing started for emailing.
21st October Beta testing complete Ver 3.0.0 uploaded
29th October minor fix added to cope with empty userid on thread creation
30th October Beta testing automatic redirection to spiders/bots IP
31st October New xml uploaded with automatic redirect to IP
25th November Minor fix for blank forumid fixed
26th November 2011 Fixed version check & create thread Off by default
17th December 2014 Version 3.1.0 uploaded, Hook changed extra logging and statistics added by Ozzy47 (Chris)
18th December 2014 Version 3.1.1 uploaded, prevented spiders being counted when mod turned off.
17th December 2014 Version 3.1.2 uploaded, due to rogue code from another mod
The Bad Bots list is now included in the product :)
Please prune out all those that you wish to be able to see your site (i suggest you definately prune out "DA" and "Custo" :

Support will now only be given to those who have this mod marked as INSTALLED

When you say "User Agents", you mean users or what? Not sure I understand purpose of this mod?

A user agent string is delivered by pretty much everyone and everything that visits your site, if you have your vbulltin options set so admin can resolve ip addresses then go to who's online then click the ip address that you see on the right for a guest, spider or forum member it will resolve to a user agent string something like the one i posted above. Remember, Google is your friend http://en.wikipedia.org/wiki/User_agent

From my understanding, Spiders are suppose to be our friends in the Forum community?

Not when there 25 of the same spiders crawling the site day after day, non-stop. And from China, to boot.

I was wondering about that Baidu Spider. What would I add to the list to get rid of them? I usually see 10 - 20 at a time.

Thanks Boofo, if you added Baiduspider then it would ban everything that has exactly that in the UA but not everything that had Baidu, so using just Baidu will ban everything that has that as part of its string so Baiduspider would also be banned.

I like the txt output. I'll have to check this out .... tagged

my htaccess :)

Code:

BrowserMatchNoCase Baiduspider bad_bot

Deny from env=bad_bot

Quote:

Originally Posted by Boofo (Post 2231643)

Baidu

I banned them at the server level. Not catering to the Chinese or Asian market and never will cater to the Chinese or Asian market so don't need them to index my site.

Works as advertised. Thanks!

Quote:

Originally Posted by Wayne Luke (Post 2232334)

I banned them at the server level. Not catering to the Chinese or Asian market and never will cater to the Chinese or Asian market so don't need them to index my site.

It's been a long time since i've seen your name on a post Wayne, nice to see you again :)

Quote:

Originally Posted by Boofo (Post 2232347)

How did you do it at the server level?

If you use cPanel you can use the IP deny manager and ban a whole block of IP's (as well as single ones)

I thought he was talking of banning spiders themselves and not IPs.

Quote:

Originally Posted by Wayne Luke (Post 2232334)

I banned them at the server level. Not catering to the Chinese or Asian market and never will cater to the Chinese or Asian market so don't need them to index my site.

By the way Wayne, a long while ago i joined vbCodex, then it went offline for a long while and when it came back up (as it is now) there's no members, posts and i'm not registered anymore - what's happening or do you have another vb coding related site?

Puts this over the header (4.1.4):

Warning: stristr() [function.stristr]: Empty delimiter in [path]/includes/class_bootstrap.php(917) : eval()'d code on line 39

Warning: stristr() [function.stristr]: Empty delimiter in [path]/includes/functions.php(7115) : eval()'d code on line 58

i have installed this mod,but can someone post a screen how or what are the right setting for this mod?

i have this spider that is filling the online page with:

119.63.196.xx and a lot of numbers to the end ,i this like other people talk in the forum,
is the " Baidu " spider...

Code:

119.63.196.57

119.63.196.45

119.63.196.14

119.63.196.13

119.63.196.40

119.63.196.79

119.63.196.76

119.63.196.49

119.63.196.102        

119.63.196.114        

119.63.196.47        

119.63.196.121        

119.63.196.116        

119.63.196.27

how to stop this...

thank you,

Quote:

Originally Posted by Conehead555 (Post 2237421)

Thats because you turned the mod on without selecting anything to ban!!!!!!! see previous posts.

Quote:

Originally Posted by bosanci28 (Post 2237657)

Code:

119.63.196.57

119.63.196.45

119.63.196.14

119.63.196.13

119.63.196.40

119.63.196.79

119.63.196.76

119.63.196.49

119.63.196.102        

119.63.196.114        

119.63.196.47        

119.63.196.121        

119.63.196.116        

119.63.196.27

how to stop this...

thank you,

Simply put Baidu in the box for banning bots :)

like this:?
see pic:

thanks

Quote:

Originally Posted by bosanci28 (Post 2237726)

like this:?
see pic:

thanks

No, you need to remove the IP's from there, this mod is not for banning IP's this one https://vborg.vbsupport.ru/showthread.php?t=268146 is!

Simply leave Baidu there and the mod will do the rest :)

Works well, finally got rid of those pesty Baidu spiders. Thanks m8.

Quote:

Originally Posted by Simon Lloyd (Post 2237770)

No, you need to remove the IP's from there, this mod is not for banning IP's this one https://vborg.vbsupport.ru/showthread.php?t=268146 is!

Simply leave Baidu there and the mod will do the rest :)

ok,done.
i also dont see any spiders from that 119.xxx ips anymore for now,bt will be checking....

thanks for your help.

installed, thank you.

Is there a comprehensive list of bad spider bots?

Some I have added to my site

Baidu
Yandex
EmailSiphon
EmailWolf
ExtractorPro
Crescent
CherryPicker
[Ww]eb[Bb]andit
WebEMailExtrac
NICErsPRO
Teleport
Zeus
Wget
LinkWalker
sitecheck.internetseer.com
ia_archiver
DIIbot
psbot
EmailCollector
nasty
verynastystuff
i-am-nasty
Twiceler

http://www.spam-whackers.com/bad.bots.htm

http://www.forumpostersunion.com/showthread.php?t=1644

very nice mod works well just what I was looking for

Modification of the Month. clicked it well worth it

click it as well very good mod could not get rid of some bots this stopped them right away thanks for this

List of bad bots added in modification description, please remember to prune out all those that you wish to be able to see your site!

Simon, it would be nice to add a text file with the bad=bots listing in a zip for the mod. That way the users of this mod have it on hand locally. ;)

Installed 4.1.5... thank you

Installed 4.1.5 PL1 Thanks.

Quote:

Originally Posted by Boofo (Post 2243835)

Simon, it would be nice to add a text file with the bad=bots listing in a zip for the mod. That way the users of this mod have it on hand locally. ;)

I'll just add another attachment, i gave the list openly like that so folk could just either copy it or pick out the ones they wanted to ban, plus its no mystery as to what they're getting but i'll definately do that tomorrow :)

the Create New Thread option does not work

I'll check that out, will post back :)

It seems that if the option for "Ban spiders in list" is checked then the spider is taken care of before the thread is created, if that option isn't checked then a thread is created, i will of course work on getting both to work together, i must have tested them one by one when creating the mod :)

thank you will be nice to see it working

Works well enough for me (killed those baidu spiders) but would be nice to see both features work together.

Quote:

Originally Posted by DaffyDuck (Post 2244644)

Works well enough for me (killed those baidu spiders) but would be nice to see both features work together.

I am sure he will get it working soon

Phrase Groups Available:

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.01764 seconds Memory Usage 1,829KB Queries Executed 10 (?)
More Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (3)bbcode_code_printable (10)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)pagenav (1)pagenav_curpage (2)pagenav_pagelink (1)pagenav_pagelinkrel (1)post_thanks_navbar_search (1)printthread (40)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start pagenav_page pagenav_complete bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages: