Ban Spiders by User Agent

[Simon Lloyd]

What this mod does
With this mod you can enter User Agents to watch or ban, you can also recieve emails or have an Output.txt created and updated with time and date of visits. It doesn't just have to be spiders, you can watch, log or ban any useragent!

How to install
Simply import the product ban_spider, the mod is active by default but none of the other options are turned on.

What is a UserAgent?
http://en.wikipedia.org/wiki/User_agent

Understanding a UserAgent string
http://user-agent-string.info/parse

Genuine User Getting Blocked?
https://vborg.vbsupport.ru/showpost.php?p=2252866&postcount=105

Tools to help
http://whatsmyuseragent.com/SwitchingUserAgents.asp
http://www.botsvsbrowsers.com/SimulateUserAgent.asp

FAQ
https://vborg.vbsupport.ru/showpost.php?p=2256512&postcount=137

What's a bot?
http://en.wikipedia.org/wiki/Spambot

How do i ban a bot?
https://vborg.vbsupport.ru/showpost.php?p=2319989&postcount=318
https://vborg.vbsupport.ru/showpost.php?p=2244937&postcount=51

Where's output.txt located?
https://vborg.vbsupport.ru/showpost.php?p=2265096&postcount=216

Bad bot lists
https://vborg.vbsupport.ru/showpost.php?p=2281880&postcount=259
https://vborg.vbsupport.ru/showpost.php?p=2265667&postcount=224
https://vborg.vbsupport.ru/showpost.php?p=2309385&postcount=281

VB4.x Version of Ban Spiders By User Agent (https://vborg.vbsupport.ru/showthread.php?t=268208)

Tested on vb3.7.x, vB3.8.x but should work on any version.

__________________________________________________ __________________
Special thanks to:
Lior
KH99
BoP5
for helping me sort out a few issues

...and beta testers
ForceHSS (Special thanks to Force for latest testing)
ozzy47
GreyHost

If you use this please mark as INSTALLED

History
9th June 2011 Orginal xml added
12th June 2011 Added both email notification and text file logging
22nd June 2011 Version 2.0.0, Added create thread on activity

Added match facility you can now use something like Yandex and it will match MOZILLA/5.0 (COMPATIBLE; YANDEXBOT/3.0; +HTTP://YANDEX.COM/BOTS)
Added clickable link to visited thread

22nd September 2011 added user redirect url selection
08th October Beta testing started for thread creation.
20th October Beta testing started for emailing.
21st October Beta testing complete Ver 3.0.0 uploaded
29th October minor fix added to cope with empty userid on thread creation
30th October Beta testing automatic redirection to spiders/bots IP
31st October New xml uploaded with automatic redirect to IP
25th November Minor fix for blank forumid fixed
26th November 2011 Fixed version check & create thread Off by default
17th December 2014 Version 3.1.0 uploaded, Extra logging and statistics added by Ozzy47 (Chris)
18th December 2014 Version 3.1.2 uploaded, due to rogue process from other mod
18th December 2014 Version 3.1.3 uploaded, due to previous one being VB4 mistakingly uploaded

The Bad Bots list is now included in the product :)
Please prune out all those that you wish to be able to see your site (i suggest you definately prune out "DA" and "Custo" :

Support will now only be given to those who have this mod marked as INSTALLED

[Simon Lloyd]

**reserved**
9th June 2011 Orginal xml added
12th June 2011 Added both email notification and text file logging
22nd June 2011 Added create thread on activity


Added match facility you can now use something like Yandex and it will match MOZILLA/5.0 (COMPATIBLE; YANDEXBOT/3.0; +HTTP://YANDEX.COM/BOTS)
Added clickable link to visited thread

26th September 2011 added user defined "Redirect to" box for adding your own url

[Aljuve]

Thank you

[Simon Lloyd]

You're welcome, please remember to mark as Installed if you have used this :)

[Lee G]

Time to be cheeky :D
Would it be possible to have this mod post a thread when you get hit by a tracked user agent?

Ie thread title user agent blah blah ip blah blah
Thread body same as title but with added suspicious behaviour / known bad agent etc

Just an idea. A lot of web site owners can take a hit. If you do a google, unless the likes of stop forum spam / project honey pot have the ips listed, there is no real other warning on the net.

What format is the test file in at present

[Simon Lloyd]

Test file? if you mean that file that is written to it's simply a text file. I'd look at the posting to a thread but to be honest you don't really want that, install the mod, add a user agent (just one) and then turn on the "write to text file" leave it on for say an hour, turn it off then check the size of the text file and the number of entries!, until the spider/bot/person get the message and remember the permanent redirect you'll get hundreds of notifications, so creating a thread or post every time would soon bloat your database for no reason.

Do you really want it?

[Simon Lloyd]

**UPDATED** Lee C's request filled :)

[Lee G]

Cheers for that Simon. Im looking for an alternative to Bad Behavior to use.
Bad behavior is still shown as being beta
Something to block specific user agents, ie Baidu, Magpie spider etc and not risk unwittingly blocking the likes of google

Top man

[appsfinder]

Hi does this stop all spider bots because i would still like to be index by google/yahoo
please can you help.

[Simon Lloyd]

It only stops those that you designate :)

[ponydaddy]

any way this will work on 4.1.5 PL1

[Simon Lloyd]

Uninstall this version and install the version for vb4.x.x https://vborg.vbsupport.ru/showthread.php?t=268208

[ForceHSS]

both of the files are the same why install the other one when both are the same?

[ponydaddy]

force is right both are the same. Any way will install the other one

[Simon Lloyd]

both of the files are the same why install the other one when both are the same?

force is right both are the same. Any way will install the other oneYes they are but vb3.x.x and vb4.x.x are different and produce different errors...etc to each other, it would be wrong to confuse a vb3 thread with answers to vb4 errors :)

[Simon Lloyd]

List of bad bots added in modification description, please remember to prune out all those that you wish to be able to see your site!

[Lee G]

Good to see this one has not been abandoned like another mod seems on here :up:
If we want to add blocks on other user agents that have spaces in them ie Windows XP, MSIE 7.0b or MSIE 5.0

Would it be a simple case of Windows\XP MSIE\5

With \ indicating the space

If anyone wonders why those three have been singles out, there is no such user agent as Windows XP, MSIE 5 is god knows how many years old and MSIE 7.0b is a beta version and windows auto updates

[Simon Lloyd]

Lee, the mod looks for a text match so "Windows\XP" will be different to "Windows XP" although i will do some testing as i've had a minor issue raised in the vb4 version that a single user can't login and gets redirected but nothing in his UA matches anything in their list, so it may be a partial match where there is a space.........i'll post back when the issue has been investigated and resolved.

[Lee G]

Been having a play and its beyond simple for banning them :up:
Windows XP is just that. Windows 3.1. 95 and ME were those endings in the user agents, but XP is a fake one. Microsoft changed how it was displayed in the user agent to Windows NT 5.1

I run a test on bots vrs browsers user agent test if a block is working
http://www.botsvsbrowsers.com/SimulateUserAgent.asp

Add the user agent to admin cp and then do a test there to see if its killed them.
In your file you have SOSO from China, they have now changed their user agent to jikespider

[Simon Lloyd]

Lee, thanks for info and the testing, so as i said "windows xp" is different to "windows\xp" in your test?

[Lee G]

I tried three variants and made sure I wernt going to be locked out the forums. Never do anything without a get out of trouble method :D

First off was two versions of xp
Windows*XP
Windows\XP

Tried viewing over at bots vrs browsers with a simple user agent of Windows XP
When neither of those two worked, I made sure I had another browser fired up, just in case I blocked myself and any browser with any mention of windows in the user agent
And tried plain old Windows XP

Worked a treat. Same as blocking Windows 95, Windows 98, Windows ME and MSIE 5
No need for any extra variables :up:

[Simon Lloyd]

...In your file you have SOSO from China, they have now changed their user agent to jikespiderI think you'll find they are still using SOSO too as right now i have 6 of them trying to access my site :)

[Simon Lloyd]

Lee, you're a star :), i just checked the UA from the vb4 version issue and it's not this mod keeping their client out of the forum, until you nudged me i'd completely forgot about bots v browsers even though i have quoted it many times here at vb.org in the past!

[Lee G]

Im just good at breaking things, ask any software developer or skin seller that I use :D

Another one to consider adding to the block list is UnwindFetchor/1.0
They are another scrape for gain outfit, by the looks of it

[Lee G]

Another one thats good for monitoring or just plain banning is this
;)
I checked it works as is. Its a give away to a messed about with user agent

Google JP must be getting busy :D

[Simon Lloyd]

Hi Lee, did you mean the wink?

[Lee G]

Should have wrapped it in php tags :D
;)

This is one example of one use agent thats been modified
Mozilla/4.0 (compatible;)

Use the tracking system and see how many user agents use it.
They think by taking out some information, you wont catch on and ban them :D

One use agent that had a bad habit of trying to inject base64 when I used the other blocking system was this one
libwww-perl

Another that only ever hits from ips on honeypot is this
w3m/

The good thing about this mod, you can track and then decide if its worth banning
Older user agents are worth watching as well. Firefox is an easy one to watch.
Firefox gives you the date the version came out

This example
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.22) Gecko/20110902 Firefox/3.6.22

If you look at the Geko/ part. It gives the date the version of firefox was released
I ban Firefox/1 and Firefox/2

Opera dont have many use it. I personally love the browser, but its another browser that auto updates. You can catch a lot of low life that way
Present version of Opera is Version/11.51

[Simon Lloyd]

Hi Guys, just to let you know there is a brand new product uploaded with everything fixed :)

[Simon Lloyd]

New product uploaded now with automatic redirect to spiders/bots own ip :)

[gigawiz]

marked as installed

I uninstalled the vb4 version and installed this one as per the instruction earlier on in the thread. This mod really is just epic and no vb forum should be without it.

EDIT - I have disabled the mod from creating new threads but it is still doing it, I have removed all the user info from the settings (user ID and forum ID) and it still posts threads which do not exist! Please help before I go totally round the bend :(

Thank you in advance.

gigawiz.

[Simon Lloyd]

is it still posting threads? i'm using the vb3.8 version my thread creation is turned off and mine isn't creating them - post back please.

[dapence]

This seems to work well! I installed, activated, and the list of 'visitors' dropped quite around 2/3.

Keep up the good work. I think the people at vBulletin proper should get you on board and have this added as a default feature of vBulletin.

Help this guy keep the agent list up-to-date.

[Boofo]

is it still posting threads? i'm using the vb3.8 version my thread creation is turned off and mine isn't creating them - post back please.

Why not wrap the whole thread creation code in a setting conditional? That way it would only run if the setting was enabled.

[Simon Lloyd]

Boofo, after my recent update it now will not fire if there isn't a forumid present, if a username is left out it automatically enters the userid 1 (as most of us start our forums off with our username) and finally the create thread option which really should have been off by default (my bad!) is now set to off by default :)

[Simon Lloyd]

This seems to work well! I installed, activated, and the list of 'visitors' dropped quite around 2/3.

Keep up the good work. I think the people at vBulletin proper should get you on board and have this added as a default feature of vBulletin.

Help this guy keep the agent list up-to-date.Glad you like it :), i doubt the guys at IB have this on their radar!

Can you mark this mod as installed please?

[Boofo]

Boofo, after my recent update it now will not fire if there isn't a forumid present, if a username is left out it automatically enters the userid 1 (as most of us start our forums off with our username) and finally the create thread option which really should have been off by default (my bad!) is now set to off by default :)

I saw that, but wrapping the code would have prevented the problem in the first place instead of having to add exit, was my point.

[Lee G]

This seems to work well! I installed, activated, and the list of 'visitors' dropped quite around 2/3.

Keep up the good work. I think the people at vBulletin proper should get you on board and have this added as a default feature of vBulletin.

Help this guy keep the agent list up-to-date.

Loads of agents to consider if you look over at vbulletin
https://www.vbulletin.com/forum/showthread.php/370200-Creating-a-vbulletin-scraper-and-spammer-trap
A good nine pages there

[Lee G]

Heres another for your low life with limited braincell capacity set forth and multiply deployment system

Firefox/3.0b5

Thats a free browser, beta version
Released: April 2, 2008

Had someone with a Romanian set of ips hit me with that

[accludetuner]

I just wanted to bump this thread and point out that this is by far one of the best and most effective anti-spam, anti-scraper, anti-harvester, anti-banned member, anti-WHATEVER mods available. It might take a little bit of time to get it setup for your specific forum, and for who you want/don't want to access your site, but once you have it setup, it's BULLETPROOF and very effective. If you're hoping to install a magical mod that magically gets rid of all the unwanted visitors from ever going to your site, keep living in magical fairy land because any effective solution is going to require some trial and error and some configuration. This mod is no exception, but once it's setup, it's absolutely fantastic!

I rated it 5 stars!

[Simon Lloyd]

Glad you like it, if you team it up with my Ban IP mod you'll get that better flexibility.

[accludetuner]

Trying the ban IP mod now.

On a side note, I help several people run other VB sites and I've been installing this spider ban mod as a default thing on all of their sites and putting in my list of user agents to ban/block. I recently installed it on a 3.6 site and it doesn't seem to be working. Is there a different hook I need to use for it to work with 3.6? Thought you might know.

EDIT: Maybe it is working and I spoke too soon. I have that pesky Baidu spider banned due to the amount of indexing it does without any regard to server resources. There were about 50 Baidu bots crawling the site but the number seems to be gradually going down (at about 20 right now). I'll keep checking it to see if Baidu disappears for good and then I'll know for sure whether it's working on 3.6 or not.

[lgpaul]

It works! Thanks!

[pitzerwm]

Thank you for creating a great mod, and thanks to Max for turning me on to it.

A question: in Who is online, in the IP column, right now I have 2 guest where the IP is totally blank. I suppose that there is an explanation for this, but is there anything that you can do about it?

[Midohash]

Fantastic ... marked as Installed ... many thanks
Just a little query! ... If I need to block unwanted visitor or member, should I put his/her IP in the spider list? ... And what is the difference between denying the IP address on cpanel or .htaccess? ... Also can I block a proxy server through this mod?

[Simon Lloyd]

For IP's you should use my other mod https://vborg.vbsupport.ru/showthread.php?t=268146 as this mod processes the useragent string and not the IP address, if you deny in cpanel or .htaccess then the user of that IP will never get to see your site or files, my mod redirects them before your site loads properly. Also with cpanel and .htaccess you have to of course log in and edit them each time you want to ban or remove a ban, mines a little more user friendly :)

[Simon Lloyd]

Thank you for creating a great mod, and thanks to Max for turning me on to it.

A question: in Who is online, in the IP column, right now I have 2 guest where the IP is totally blank. I suppose that there is an explanation for this, but is there anything that you can do about it?

Sorry, again you need this https://vborg.vbsupport.ru/showthread.php?t=268146 for IP's, this deals with the user agent string not IP's, if their IP is blank then they are cloaking it somehow, if you choose to see the User Agent (via the dropdown at the bottom of the WHo's Online window you can copy their user agent string, pop it in the list to ban and then watch them disappear :)

[Midohash]

For IP's you should use my other mod https://vborg.vbsupport.ru/showthread.php?t=268146 as this mod processes the useragent string and not the IP address, if you deny in cpanel or .htaccess then the user of that IP will never get to see your site or files, my mod redirects them before your site loads properly. Also with cpanel and .htaccess you have to of course log in and edit them each time you want to ban or remove a ban, mines a little more user friendly :)

Thanks Simon :up:, regarding your other mod is it possible to be modified in the future to block users from anonymous proxy? :confused: ... That will be a great add on :cool: ... However I note a significant drop in the number of my site visitors after installing the mod, is that attributed to blocking bad bots only or could be also some normal visitors are not able now to access my forum? ... Also is the drop in the visitor numbers would affect my over all alexa rank? ... Regards,

[pitzerwm]

I installed this and with Ban IPs apparently have removed a large percentage of my issues.

I was wondering if I get this as the "guest" dhcp-0-24-b2-58-41-4a.cpe.eaglecable.net

Can I put this total "agent" in, and it will ban this person?

Thanks

[photonetau]

I installed this about 9 hours ago and baidu is still hitting at the rate of 200 hits an hour is that usual , how long before they give up?
BTW they are redirected to the default site.

[Simon Lloyd]

Thanks Simon :up:, regarding your other mod is it possible to be modified in the future to block users from anonymous proxy? :confused: ... That will be a great add on :cool: ... However I note a significant drop in the number of my site visitors after installing the mod, is that attributed to blocking bad bots only or could be also some normal visitors are not able now to access my forum? ... Also is the drop in the visitor numbers would affect my over all alexa rank? ... Regards,It could be modified to ban those without IP and this one could be modified to ban those without user agent but would need a lot of testing!

Yes, your visitor drop is due tothe banning, you'll find (especially if you installed the spiders on forumhome mod) that most of the visitors were bots in the list like Baiduspider, they're insatiable. Bots don't make up part of your alexa ranking, believe it or not Alexa makes it's assumptions from the amount of visitors to your site that have the Alexa toolbar installed :)

I installed this and with Ban IPs apparently have removed a large percentage of my issues.

I was wondering if I get this as the "guest" dhcp-0-24-b2-58-41-4a.cpe.eaglecable.net

Can I put this total "agent" in, and it will ban this person?

ThanksYes of course, the exact agent string will help zero in on that person/bot...etc, i added partial matching because of bots changing their UAs..etc but this mod was always built on using the entire string :)

I installed this about 9 hours ago and baidu is still hitting at the rate of 200 hits an hour is that usual , how long before they give up?
BTW they are redirected to the default site.I guess you are still seeing Baidu because you are using Paul M's who visited mod, both his mod and mine are doing their job, his picks them up because they make a direct call to a thread or post...etc and get logged but as soon as they attempt to access the thread or post they are redirected to whichever way you choose with a 301 redirect header :)

If you are seeing Baiduspider and not using Paul's mod then pm me your site url and access details and we'll get you sorted!

[Midohash]

Thanks a lot Simon :up: ... I still have same problem with Baidu :( ... I have checked my raw access log today and still having loads of Baidu spiders in my site 48 hours after installing the mod :eek: ... this is how they appear in my log file:

Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Any recommendations?

Regarding Alexa, I have certified my site and have pasted their javascript code in the header of my forum as they recommend so they can analyse my website traffic directly regardless of visitors with/without alexa toolbar ... I don't know if my alexa rank will drop due to drop in the overall visitor numbers after banning spiders?

Regards

[Simon Lloyd]

of course your raw logs from cpanel, plesk or whathaveyou will show them visiting your site, remember this mod redirects them when trying to load your site, it does that regardless of whether they are trying to load the homepage or a thread, your Alexa shouldn't drop on account of redirecting spiders, in fact i've been redirecting these spiders for ages on one of my sites that was Page Rank 2 and it's now PR3, so if it's not hurting Google rank it has to be good :)

Can you give me a screenshot of your settings for ths mod?

[Midohash]

Thanks Simon, enclosed is a screenshot of my settings for the mod. My Alexa rank dropped from 83000 to 84000 but not sure if that related to the drop in the visitors count after applying the mod or just an incidental finding :confused:

http://img29.imageshack.us/img29/8616/spidersq.jpg

[Simon Lloyd]

Midohash, if you want to pm me admin access i'll take a look.

[Doug Nelson]

Excellent mod, thank you.

What is the advantage/disadvantage to redirecting spiders back to their own IP?

[Simon Lloyd]

Excellent mod, thank you.

What is the advantage/disadvantage to redirecting spiders back to their own IP?You have to send them somewhere and whilst the example i gave you is a fun place to send them and they expect a certain amount of bots there's no guarantees that it will always be there, however, if you redirect to their own IP they will always have somewhere to be redirected to :)

[gsmlover4u]

to block spiders are benificial or not ?

[Simon Lloyd]

Blocking spiders will save you tons on your bandwidth and server load, take the Baiduspider they send hundreds upon hindreds of bots to index your site and they don't obey robots.txt

[gsmlover4u]

working on vbulletin 4.1.10

[steviewonder44]

The ips for baidu I also ban: 123, 124 125 and 58

[Simon Lloyd]

As you've downloaded thsi could you mark it as installed?
This particular mod is for banning User Agents, i do have another that's specifically for IP's https://vborg.vbsupport.ru/showthread.php?t=268146

Keep an eye out for my upcoming geo-banning of users :)

[Midohash]

Keep an eye out for my upcoming geo-banning of users :)

That will be another great mod Simon :up:, can I add this phrase to the list of user agents: Baiduspider/2.0 or just Baidu or Baiduspider?

[Simon Lloyd]

simply Baidu will do it :)

I'm curently working on geo-banning and geo-targeting for ads (so you can maximise your revenue).

[Midohash]

simply Baidu will do it :)

I'm curently working on geo-banning and geo-targeting for ads (so you can maximise your revenue).

Superb :up: ... does that mean I can allow traffic from some countries to google adsense ads and block traffic to ads from other countries with low click revenue? :confused:

[Simon Lloyd]

When i release it you will be able to target one country with ad_navbar_below and ad_ foorter everyone else will see what they always have seen!, thats the lite version, the full version will cater for multiple countries where you can designate ads to be shown in any current vbulletin ad location :)

[lebowski99]

Simon, I have a dumb question for you. I installed your mod but when I go into the "Who's Online" page I can still see bots hammering away at restricted areas in my site. How do I go about adding the user agent to the spider list on your add-on control panel? I figured out how to display the full user agent string. For example, I have one right now that reports:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts; .NET CLR 1.1.4322; PeoplePal

Looking at this string, what do I enter into the Spider List in the Ban Spider settings?

[Simon Lloyd]

Just enter funwebproducts :)
You can use the entire string if you wish or even peoplepal, you will see the bots who are in the banned list for anything up to an hour after install and you may on the odd occassion see them in the list but rest assured that they are being redirected, if you notice them there all the time still then pm me admin access to your site with permissions and i'll sort that for you :)

See the pic for adding to the list :)

136597

[lebowski99]

Just enter funwebproducts :)
You can use the entire string if you wish or even peoplepal, you will see the bots who are in the banned list for anything up to an hour after install and you may on the odd occassion see them in the list but rest assured that they are being redirected, if you notice them there all the time still then pm me admin access to your site with permissions and i'll sort that for you :)

See the pic for adding to the list :)

136597

Sounds easy enough. But here is one from russia and the user agent is listed simply as:

Opera/9.00 (Windows NT 5.1; U; en)

What do I enter for this one? Seems like if I enter the entire string, it would potentially block a legit user.

[Simon Lloyd]

Nope :) enter the entire string!, if by some very remote chance you find you've blocked a legit user then let me know and we'll work something out.

[trotskid]

Installed, thanks!

[Simon Lloyd]

You're welcome :)

[Lee G]

Dont know if its listed in your bad user agent list, one of the worst I have found is OMGILI
User agent
omgilibot/0.3 +http://www.omgili.com/Crawler.html

If you right click on any page on the search engine site, you will see that all external links are no follow

They were hitting me for over 200 pages a day until I put an ip ban on them

[Simon Lloyd]

Dont know if its listed in your bad user agent list, one of the worst I have found is OMGILI
User agent
omgilibot/0.3 +http://www.omgili.com/Crawler.html

If you right click on any page on the search engine site, you will see that all external links are no follow

They were hitting me for over 200 pages a day until I put an ip ban on themHmm, it's not added there but folk can tailor their own, i only gave a starting base, i'm sure many have now customised it ;)

I can bet that omgili aren't as bad as Baidu :)

[Lee G]

For some reason I get zero baidu hits :D
Either cloudflare block them or they take note of robots txt
Putting ip bans into the cf control panel saves killing my own server with htaccess bans

There seems to be a mass of junk with no user agent hitting from AmazonAWS at present
I need to find out if anything good comes from their servers, then consider a complete ban on all their ips. Any user agent being tracked is littered with AmazonAWS ips

[Max Taxable]

For some reason I get zero baidu hits :D
Either cloudflare block them or they take note of robots txt
Putting ip bans into the cf control panel saves killing my own server with htaccess bans

There seems to be a mass of junk with no user agent hitting from AmazonAWS at present
I need to find out if anything good comes from their servers, then consider a complete ban on all their ips. Any user agent being tracked is littered with AmazonAWS ipsNeither happens. I have CF and robots.txt.

AmazonAWS is a friendly spider, from what I've seen over the years.

And Simon - here I am, marked installed.

[Lee G]

AmazonAWS cloud hosting

Full range of AmazonAWS ip's
http://www.forumpostersunion.com/showthread.php?t=10490

Some of the bad bots being run out of there
http://www.webmasterworld.com/search_engine_spiders/3828718.htm

At least one bot owned by Google also operates out of the AmazonAWS ip ranges
Postrank were brought out by Google
http://www.postrank.com/

[final kaoss]

I'll give it a shot to scare baidu away

[Max Taxable]

My current (updated) list of banned user agents entered into this Mod:

baiduspider
beta.statsit.com
statsit
SiteIntel
Yandex
GomezAgent
FunWebProducts
MSIE 1
MSIE 2
MSIE 3
MSIE 4
MSIE 5
MSIE 6
Nesotebot
DCPbot
Opera/1
Opera/2
Opera/3
Opera/4
Opera/5
Opera/6
Opera/7
Opera/8
AOL Advertising R&D
DataCha0s
aiHitBot
Apache-HttpClient
Zend_Http_Client
ReverseGet

Entering the older MSIEs and the older Operas has virtually eliminated bot registration attempts. It's down to just one or two a day, and the "IsBot" Mod is still catching those. It used to be, 40-50 a day that would be caught.

[LouiseWilson]

Installed and working with 4.2.0 P2

thank you for the modification

[Max Taxable]

Hey Simon.... When this Mod is set to create threads, could it be updated to also include the IP of the bot in the post?

[Simon Lloyd]

Sure, are you sure you want to create threads? you'd get soooooo many of them, remember i added the thread making and logging for test purposes so folk could see that it was working for their selected spiders are being dealt with.

[Max Taxable]

Sure, are you sure you want to create threads? you'd get soooooo many of them, remember i added the thread making and logging for test purposes so folk could see that it was working for their selected spiders are being dealt with.Yes I have had this Mod working for so long, the really bad spiders like Baidu have stopped visiting completely. This Mod generated only 88 threads yesterday. I plan to turn the thread making feature on and off periodically, just for occasional monitoring purposes.

The IPs in the post would be great, if it's not too difficult and if you please.

Edit: It would be nice too if we could make it post everything in the same thread in addition to the option of creating a new topic for each bot.

[Simon Lloyd]

Ok, if you can wait until next week as i've promised my wife that i'll give the computer a rest for a week (other than important stuff) so i'll get to it after then :)

[Simon Lloyd]

@Max Taxable for now (just because i can't make you wait :)) go to admincp>plugins & products>Plugin Manager find the Ban Spiders mod's plugins and look for Post UA Activity to thread and replace all of the code you see there for thisif ($vbulletin->options['bsactive'] AND $vbulletin->options['bs_report_createthread'])
{
$agentip = strtoupper($_SERVER['REMOTE_ADDR']);
$DateOfRequest = date('m-d-Y H:i:s');
$user_agent1 = strtoupper($_SERVER['HTTP_USER_AGENT']);
$bots1 = explode("\r\n", $vbulletin->options['bs_sel_list']);
$n = sizeof($bots1);
for ($i=0;$i<$n;$i++) {
if (stristr($user_agent1,$bots1[$i]))
{
// Backend Files
require_once('./global.php');
require_once('./includes/class_dm.php');
require_once('./includes/class_dm_threadpost.php');
require_once('./includes/functions_databuild.php');

$threaddm =& datamanager_init('Thread_FirstPost', $vbulletin, ERRTYPE_ARRAY, 'threadpost');
$threadinfo = array();
$vboptions =& $vbulletin->options;
$username = $vbulletin->userinfo['username'];
$forumid = $vbulletin->options['bs_report_forumid'];
$foruminfo = fetch_foruminfo($foruminfo['forumid']);
$title = "Activity from Bot No. $i ($bots1[$i]) in your list";
$userid = $vbulletin->options['bs_report_userthread'];
if ($userid == '')
{
$userid=1;
}
if ($forumid =='')
{
exit;
}
$pagetext = "Date and Time: $DateOfRequest\nAssociated Username (if any): $username\nMatched bots[$i]: $bots1[$i] \nWith User Agent: $user_agent1\n\nTheir IP:$agentip\n\nThey visited:\n$_SERVER[SERVER_NAME]$_SERVER[REQUEST_URI]\n\n";
$posticon = $vbulletin->options['subthread_posticon'];
$allowsmilie = '1';
$visible = '1';

$threaddm->set_info('forum', $foruminfo);
$threaddm->set_info('thread', $threadinfo);
$threaddm->setr('forumid', $forumid);
$threaddm->setr('userid', $userid);
$threaddm->setr('pagetext', $pagetext);
$threaddm->setr('title', $title);
$threaddm->set('iconid', $posticon);
$threaddm->set('allowsmilie', $allowsmilie);
$threaddm->set('visible', $visible);

// Lets see what happens if we save the page
$threaddm->pre_save();
if(count($threaddm->errors) < 1) {
// Basically if the page will save without errors then let do it for real this time
$threadid = $threaddm->save();
unset($threaddm);
build_thread_counters($threaddm);
build_forum_counters($forumid);
build_forum_counters($threadinfo['forumid']);
// unset($threaddm);
} else {
// There was errors in the practice run, so lets display them
var_dump ($threaddm->errors);
}

}
}
}You'll now get their ip in the thread thats created :)

The posting to same thread for each bot will require queries to the database, we'd have to add a column to store thread id's against bot names...etc, it could work out kinda messy and resource expensive, but, i'll still look in to it!

[Max Taxable]

Thanks Simon! I was content to wait though.:D

[ChiNa]

Hi Simon, Can you maybe give us a link to some newer bad behavior bots.
I did check your links for the Bad Bots, I just dont know if you already have banned those bots in the links, and how to bann a bot using a webhost, which says "dreamhost.com" fake google?
But it will not make problems for me to bann a REAL hosting company?

Waiting for a reply befor I install it! Thank you

[Simon Lloyd]

Read the links properly, i have linked you to the "how do i ban a bot?" and links to some maintained lists, and as i have said repeatedly banning bots/spiders/UAs is a personal thing :)

[Max Taxable]

Simon...

Is there a quick line of code we can add that captures User Agent info as well as IP, and includes this info in the email?

[Simon Lloyd]

Isn't it already included in an email?

This will show the useragent:
With User Agent: $user_agent1\n\

Add it anywhere under Matched Bots in the email message, should be good, i dont have time right now to check it properly as i've been up 21 hours now and desperately need sleep!

[Max Taxable]

Isn't it already included in an email?

This will show the useragent:
With User Agent: $user_agent1\n\

Add it anywhere under Matched Bots in the email message, should be good, i dont have time right now to check it properly as i've been up 21 hours now and desperately need sleep!It's not but I'll play around with this and let you know what works. Thanks for the quick response.

[Simon Lloyd]

Just checked the code and it should give you the UA in the mail, here's the code that does it$message = "DATE: " .$DateOfRequest ."\n\nUserAgent: " .$_SERVER['HTTP_USER_AGENT'] . " \n\nhas visited \n\n" . $vbulletin->options['homeurl']
. $_SERVER['REQUEST_URI'] . "\n\nFound In Your List As " . $bots2[$iz] . "\n\nRegards, \n\nBan Spider Mod";see the bit that says\n\nUserAgent: " .$_SERVER['HTTP_USER_AGENT'] . "that should show the useragent!?

[Max Taxable]

Just checked the code and it should give you the UA in the mail, here's the code that does itYes, I'm dumb. *slaps forehead* I apologize for wasting your time, this isn't the Mod I am looking to get this feature on. I thought this was the thread for the "Is Bot" mod (https://vborg.vbsupport.ru/showthread.php?t=135094), which doesn't capture the UA.

That mod is not supported and the developer hasn't been online for over a year. Any chance we can add this feature to that one, working via PM?

[Simon Lloyd]

Ok i've pm'd you witb what i think is a solution for that mod :)

[Midohash]

Hi Simon,

I would much appreciate it having a look on my recent thread regarding an evil obscene banned member keep registering himself in dozens of evil names every day to cause as much damage as possible to the forum:

https://vborg.vbsupport.ru/showthread.php?t=289646

I have his user agent strings but afraid to block other innocent members if putting the full strings in Ban Spiders by User Agent product! Would much appreciate your advise.

Mozilla/5.0 (Windows NT 6.0; rv:16.0) Gecko/20100101 Firefox/16.0

and

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0)

Many thanks,

[Max Taxable]

Hi Simon,

I would much appreciate it having a look on my recent thread regarding an evil obscene banned member keep registering himself in dozens of evil names every day to cause as much damage as possible to the forum:

https://vborg.vbsupport.ru/showthread.php?t=289646

I have his user agent strings but afraid to block other innocent members if putting the full strings in Ban Spiders by User Agent product! Would much appreciate your advise.

Mozilla/5.0 (Windows NT 6.0; rv:16.0) Gecko/20100101 Firefox/16.0

and

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0)

Many thanks,There's nothing in those strings you can block without affecting 100s of millions of people.

[Midohash]

There's nothing in those strings you can block without affecting 100s of millions of people.

ٍBelieve it or not! ... I put the full 2 strings in ban spiders by user agent and in spite of that he had registered again by another evil username!!! ... Surprisingly when I tried to check his user agent for his newly registered account while he is still online I found it obscene words rather than:

Mozilla/5.0 (Windows NT 6.0; rv:16.0) Gecko/20100101 Firefox/16.0


!!!!!!!!

He replaced his user agent string by obscene words!

[Midohash]

Just to let you know that the evil obscene guy I am talking about him has registered here by my real name (mohamed hashesh) to continue his disgraceful and obscene behavior!

[Max Taxable]

Just to let you know that the evil obscene guy I am talking about him has registered here by my real name (mohamed hashesh) to continue his disgraceful and obscene behavior!So, he is watching you jump through his hoops here, and knows how to spoof his user agent string. (Which, isn't all that hard to do.)

You are giving him just what he wants, attention.

[Midohash]

So, he is watching you jump through his hoops here, and knows how to spoof his user agent string. (Which, isn't all that hard to do.)

You are giving him just what he wants, attention.

I have no doubt he is Psycho! ... He spends at least 18 hours watching the forum, just waiting for the moment registration to new members opens again to bombard the forum with dozens of evil usernames! ... Would appreciate it communicating via PM if you have any suggestions

Many thanks

[Simon Lloyd]

Midohash, i'll pm you later with some other way of helping against this.

[Midohash]

Midohash, i'll pm you later with some other way of helping against this.

Thanks a lot Simon, I have received your PM and sent a reply

[Simon Lloyd]

I'm looking for feedback guys!
Would it be beneficial to automatically ban bots that exceed x number of bots at any one time?

So, the likes of Baiduspider send around 200 at any one time, so if i entered say 150 (in place of x) in a settings box then they would automatically get added to the ban list, let me know your views as i'm not going to work on something nobody feels is needed :)

[Midohash]

I'm looking for feedback guys!
Would it be beneficial to automatically ban bots that exceed x number of bots at any one time?

So, the likes of Baiduspider send around 200 at any one time, so if i entered say 150 (in place of x) in a settings box then they would automatically get added to the ban list, let me know your views as i'm not going to work on something nobody feels is needed :)

Your mod is highly needed Simon and has helped me significantly when I was on shared hosting in reducing the server load and saving bandwidth when bad bots used to hit heavily and unnecessarily my site :up:

However I have two queries here:

1- How the mod would differentiate between good bots like Google for instance and bad bots if they send higher than the threshold you have already set?

2- What is the difference between banning bad bots through this mod and that through the .htaccess?

Thank you

Mido

[Simon Lloyd]

Ok, firstly i'd incorporate a safe bot list where you can add bots that you dont want to be banned. The difference with this and .htaccess is, with .htaccess the bots don't even get to call any page on your site but having a lot in your .htaccess list can cause your server to load and slow down, this mod allows the call of a page...etc but as soon as they do it never completes for them as they are then redirected straight away and as you've experienced they quickly drop off and so does your server load :) and you save tons of bandwidth.

[Midohash]

Ok, firstly i'd incorporate a safe bot list where you can add bots that you dont want to be banned. The difference with this and .htaccess is, with .htaccess the bots don't even get to call any page on your site but having a lot in your .htaccess list can cause your server to load and slow down, this mod allows the call of a page...etc but as soon as they do it never completes for them as they are then redirected straight away and as you've experienced they quickly drop off and so does your server load :) and you save tons of bandwidth.

Good idea :up: ... I didn't know that blocking bad bots through .htaccess could increase the server load ... I thought the reverse! ... Thanks for the explanation ... When do you expect the new modifications in your mod to take place?

[Simon Lloyd]

Well im waiting on more votes from people saying that they'd think it was a good idea and use it so give it a week or two.

[Midohash]

Well im waiting on more votes from people saying that they'd think it was a good idea and use it so give it a week or two.

OK I'll wait for the new modifications ... Just a quick question ... Can I use this phrase to ban the bad bot in your mod?

RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [OR]

Or I have to change it to craftbot alone?

Thank you

[Simon Lloyd]

You need to understand how the mod works, take a look here https://vborg.vbsupport.ru/showpost.php?p=2377564&postcount=381 should explain it for you.

[Midohash]

You need to understand how the mod works, take a look here https://vborg.vbsupport.ru/showpost.php?p=2377564&postcount=381 should explain it for you.

Thanks, I got the point meaning!

[hotshot]

Isn't this supposed to reduce the number of Bot registations too? I have this mod installed and I still get hundreds of bot registrations that are blocked by vbStopForumSpam.

[Max Taxable]

Isn't this supposed to reduce the number of Bot registations too? I have this mod installed and I still get hundreds of bot registrations that are blocked by vbStopForumSpam.This has nothing whatsoever to do with spam bot registrations. This Mod blocks by USER AGENT STRINGS you define. But be careful - it doesn't know if it's a bot or a human, and doesn't care - so don't enter anything that is a common UA term. You have the potential to block billions of legitimate users!

This however, does have to do with spam bot registrations:

https://vborg.vbsupport.ru/showthread.php?t=289463

[valdet]

The output.txt is generated as bots found in your list attempt to call a forum or thread, there's no time lag and the file should be created straight away. If you have no cms then the file should be available at www.mysite.com/output.txt if forum is in a folder then something like www.mysite.com/forum/output.txt

Any issues post back and i'll deal with them for you :)

Hi Simon,


Is there any particular server setting that needs to be enabled for the Write to output.txt file to work?

I have set the settings to create a new thread and store entries in log file (I switched between them as well. Enabled one, disabled second and vice-versa)

I am on vB 3.8.7
None are working.

Thanks,

[Simon Lloyd]

It will be to do with whether you have permission to write to the file on your server, the script creates one if it doesn't exist and then writes to it , if I remember rightly at /public_html level, i'll check tomorrow when I get home and give you a "How to" write at that level if my mod writes above it.

[Max Taxable]

If you haven't added "rippers" to your list, do so.

[valdet]

It will be to do with whether you have permission to write to the file on your server, the script creates one if it doesn't exist and then writes to it , if I remember rightly at /public_html level, i'll check tomorrow when I get home and give you a "How to" write at that level if my mod writes above it.

Thanks for reply.

The file is created and has proper CHMOD permissions. But neither log file option, nor the Create a thread option are working.
FYI, I am using my admin account to create threads into a not public forum. This forum allows admins to post new threads.

Thanks.

[Simon Lloyd]

That's not what I meant, I mean you may not have permissions on your server to write a file above the /public_html folder (the log file is written to the server not the forum!). When I sort out the issues with my forum later i'll take a look at this and release a fix :)

[tambo]

Ok, firstly i'd incorporate a safe bot list where you can add bots that you dont want to be banned. The difference with this and .htaccess is, with .htaccess the bots don't even get to call any page on your site but having a lot in your .htaccess list can cause your server to load and slow down, this mod allows the call of a page...etc but as soon as they do it never completes for them as they are then redirected straight away and as you've experienced they quickly drop off and so does your server load :) and you save tons of bandwidth.
Consider me a +1 vote.

I still get great use out of this and, honestly, anything that can make a plugin dynamic with a degree of self-sufficiency is right up my street.

So... yes!

[Simon Lloyd]

I'd forgotten all about that as i didnt get much response, if i have time over the next week or so i'll look at doing that :)

[Dan49]

Thank you for the mod Simon.My current (updated) list of banned user agents entered into this Mod:

MSIE 1
MSIE 2
MSIE 3
MSIE 4
MSIE 5
MSIE 6

Entering the older MSIEs and the older Operas has virtually eliminated bot registration attempts. It's down to just one or two a day, and the "IsBot" Mod is still catching those. It used to be, 40-50 a day that would be caught.I have a bunch of guests with Russian IPs with MSIE 7 & MSIE 8 in the UA. It's been a couple of years since Max's list, is it now safe to block 7 & 8?

[Simon Lloyd]

I know I sound like a broken record but it really is a personal choice about who you want to block, less fortunate countries will have older browsers and it's probably safe to block up to MSIE6, later browsers will still be in use, so you really need to check your demographics and make sure you are catering for your intended audience.

Sorry but that's the best advice I can give.

[Dan49]

What does MSIE 7 stand for?
What about Mozilla/4.0?

Is there a list somewhere that explains the meaning?

[Simon Lloyd]

Microsoft Internet Explorer
Mozilla is firefox, use google to find out what these terms are if you are unsure, there's not a list that I know of or have even searched for but you can use the tools I've shown in the first post to help you find some info around these :)

[Max Taxable]

I know I sound like a broken record but it really is a personal choice about who you want to block, less fortunate countries will have older browsers and it's probably safe to block up to MSIE6, later browsers will still be in use, so you really need to check your demographics and make sure you are catering for your intended audience.

Sorry but that's the best advice I can give.That's the main point really.

[Max Taxable]

What does MSIE 7 stand for?
What about Mozilla/4.0?

Is there a list somewhere that explains the meaning?Do NOT enter the term "Mozilla" into this mod! You will be blocking just about every windows computer on the planet if you do!

Here's a fairly typical Windows computer user agent string:

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727

ALL windows computer user agent strings are going to have the word "Mozilla" in them no matter what browser the computer actually uses.

[Dan49]

Entering Mozilla/4.0 is different than entering Mozilla correct? If so, is blocking Mozilla/4.0 unreasonable? All the Russian IPs are using it and no one else.

[Max Taxable]

Entering Mozilla/4.0 is different than entering Mozilla correct? If so, is blocking Mozilla/4.0 unreasonable? All the Russian IPs are using it and no one else.NO Mozilla AT ALL!

Most ALL windows computers gonna have that "Mozilla4.0" yours probably does too! Have you studied your OWN user agent string?

[Dan49]

But Simon's original list of UA that comes with the mod contains:

Mozilla.*Indy
Mozilla.*NEWT
Mozilla*MSIECrawler

And my understanding is that will only block those particular strings? It looks like this is true because they're on my list and there are plenty of Mozilla/5.0 and Mozilla/4.0 in my forum.

So if I understand correctly; adding Mozilla/4.0 (which I will not do as per your recommendation) will not affect Mozilla /5.0?

[Max Taxable]

But Simon's original list of UA that comes with the mod contains:

Mozilla.*Indy
Mozilla.*NEWT
Mozilla*MSIECrawler

And my understanding is that will only block those particular strings? It looks like this is true because there are plenty of Mozilla/5.0 and Mozilla/4.0 in my forum.I never saw that on his list or any other list, but yes as long as "mozilla" isn't on a line by itself you got no problems.

But what i am telling you is, blocking "Mozilla/4.0" gonna block billions of computers, not just the Russian ones.

Find something unique in their strings to block, if you want to use this to block them. You might not find any such.

[Dan49]

Understood, thanks.

[Dan49]

Max, you may be interested in this. I had a legit user blocked out of my forum. I checked and rechecked his UA and could not figure out why he was blocked Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) Of course turning off the mod unblocked him, so I knew it had to be something.

On a hunch I looked at the MSIE entries I copied from your old post:

MSIE 1
MSIE 2
MSIE 3
MSIE 4
MSIE 5
MSIE 6

I realized that MSIE 1 is the culprit being the first part of MSIE 10.0;

Have you since removed MSIE 1 from your list?

Is there another way to make MSIE 1 work without affecting MSIE 10? As well as the other MSIE, like MSIE 2 for when IE gets to version 20 :)?

[Simon Lloyd]

I doubt there's a pc on the planet that still uses MSIE 1,2 or 3 :)

Remember this mod was built to ban spiders or bots which usually display their bot name in their UA, using the mod outside of that scope is bound to take you in to some grey areas, make sure you use the tools shown in the original post of this thread to check new UA's before you ban them.

[Dan49]

Is there a way to block MSIE 1 without affecting MSIE 10?

[Simon Lloyd]

All you can do is add a space after it, but again all it will do is look for MSIE 1_ where _ is the space (it doesn't physically look like _ that was for illustration purposes), but if it doesn't appear in the UA with a space after it, it wont ban it.

[Dan49]

Thanks.

Another general question. I understand that malicious bots can fake the UA. Why don't they all fake it to display as Google? This way anybody would be hesitant to block them? BTW I do use your IP ban mod also.

[Max Taxable]

Max, you may be interested in this. I had a legit user blocked out of my forum. I checked and rechecked his UA and could not figure out why he was blocked Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) Of course turning off the mod unblocked him, so I knew it had to be something.

On a hunch I looked at the MSIE entries I copied from your old post:

MSIE 1
MSIE 2
MSIE 3
MSIE 4
MSIE 5
MSIE 6

I realized that MSIE 1 is the culprit being the first part of MSIE 10.0;

Have you since removed MSIE 1 from your list?

Is there another way to make MSIE 1 work without affecting MSIE 10? As well as the other MSIE, like MSIE 2 for when IE gets to version 20 :)?Yes this was covered earlier in the thread, and have since removed MSIE 1 from my list. This was months ago. Like Simon said, the number of surviving computers on the planet still using MSIE1 is so tiny, it's not worth blocking.

[Max Taxable]

Thanks.

Another general question. I understand that malicious bots can fake the UA. Why don't they all fake it to display as Google? This way anybody would be hesitant to block them? BTW I do use your IP ban mod also.Some of them do so, but it's very easy to spot those fakes since Google only uses one or two IP addresses. I am pretty sure there's a Mod for blocking spoofed google UA's somewhere.

[Dan49]

I found this information about verifying Google bot (https://support.google.com/webmasters/answer/80553?hl=en). And used this tool http://ipadmin.junkemailfilter.com/rdns.php. Is this the best way? Or is there a list of the IP addresses google uses?

I'd appreciate a link to the mod you mentioned, I searched and can't find it.

[valdet]

Hi Simon,

Hope you got your forum problems sorted :)

I noticed that the mod will not create new threads on private forums, where only admins can post/read.
I am using my admin account as thread creator, so it has permissions to create threads. The threads are created only on public forums where other members and guests can see them, which of course isn't nice.

The output.txt file writing is also not logging anything.

But from my server logs, I see this mod has been doing incredibly well in blocking some nasty spiders (mainly Magpie and Spinn3r).

For those who don't check their server logs, I would advise to check them and see if Magpie is sucking up to 40% of bandwidth as it was until recently in one of my sites.

[Max Taxable]

I found this information about verifying Google bot (https://support.google.com/webmasters/answer/80553?hl=en). And used this tool http://ipadmin.junkemailfilter.com/rdns.php. Is this the best way? Or is there a list of the IP addresses google uses?

I'd appreciate a link to the mod you mentioned, I searched and can't find it.I only vaguely remember seeing such, and it was sometime back. And I might even be wrong.

[Simon Lloyd]

Hi Simon,

Hope you got your forum problems sorted :)

I noticed that the mod will not create new threads on private forums, where only admins can post/read.
I am using my admin account as thread creator, so it has permissions to create threads. The threads are created only on public forums where other members and guests can see them, which of course isn't nice.

The output.txt file writing is also not logging anything.

But from my server logs, I see this mod has been doing incredibly well in blocking some nasty spiders (mainly Magpie and Spinn3r).

For those who don't check their server logs, I would advise to check them and see if Magpie is sucking up to 40% of bandwidth as it was until recently in one of my sites.Glad it's working for you, on my forum I simply added a forum where only staf had permissions to view and made the threads there, I also had no issue with logging.

I did get my forum issues sorted, turns out it was a half assed script kiddie attempt to kill it but got it sorted (one of my forums allowed html in the post environment....really bad idea!). When I get chance (which will be after Thursday as my wife is having an op) I will retest the mod from scratch and post specifics here.

[jl255]

i suppose it shld be pretty safe to use the default list of spider list to ban on this plugin? i've no idea which to prune.....

[Simon Lloyd]

There are quite a few lists posted throughout this thread, take a look at them, if you use the built in one thenPlease prune out all those that you wish to be able to see your site (i suggest you definately prune out "DA" and "Custo" :

[Max Taxable]

i suppose it shld be pretty safe to use the default list of spider list to ban on this plugin? i've no idea which to prune.....See if the default one helps the issue you're having, first. Is my suggestion. The default list is a good list.

[Wajow-community]

Is there a mod for vb 4.x.x

[Max Taxable]

Found on the developer's profile:

https://vborg.vbsupport.ru/showthread.php?t=268208

[qpurser]

Really love this mod a lot and works great on 4.2.1

Finally Baidu is not showing up anymore in my "online users" list.

I saw another bot recently searching my forum and from some research it seems to be a bad one also: AhrefsBot http://blocklistpro.com/content-scrapers/ahrefsbot-seo-spybots.html

I added this to my list:
Mozilla/5.0 (compatible; AhrefsBot/2.0; +http://ahrefs.com/robot/)
and
Mozilla*Ahref

Was this the correct way to do it?

[Max Taxable]

I added this to my list:
Mozilla/5.0 (compatible; AhrefsBot/2.0; +http://ahrefs.com/robot/)
and
Mozilla*Ahref

Was this the correct way to do it?NO!!!



Including "Mozilla" and "compatible" in this list blocks just about the entire world!

Get rid of that entry and simply put: "AhrefsBot" in instead!

[qpurser]

NO!!!

Including "Mozilla" and "compatible" in this list blocks just about the entire world!
Get rid of that entry and simply put: "AhrefsBot" in instead!

Thank you for the quick reply.
I was reading post #127 and 128 and thought as long there is something behind the "mozilla" in the same line there wouldn't be an issue. My mistake I guess

[Max Taxable]

Thank you for the quick reply.
I was reading post #127 and 128 and thought as long there is something behind the "mozilla" in the same line there wouldn't be an issue. My mistake I guessYou might be right but 1.) I wouldn't risk it, and 2.) it's not necessary.

[Simon Lloyd]

When banning whatever you add to the list will be looked for in its entirety, so if you enter "today" then it will ban:
Today
thisdayistoday
wheretodayis
one day today

if you entered "here today" then it will not ban:
one day today
thisdaytoday
today....etc

but it will ban:
we were here today
allhere todayagain....etc

When banning bots make sure you go to WOL and check out their UA as it may not contain their name in the UA.

[DemOnstar]

Had this mod for a while now, forgot to mark installed..

Cheers, does the trick...

[Simon Lloyd]

Thanks for the feedback! :)

[waterart]

ok not sure what i am missing here but i dont see the screenshots that others are showing where it lists the area of setting the spiders to block etc... where do i go in the admin to manage this mod?

[Simon Lloyd]

Go to ban spiders by user agent in vbulletin options :)

And the pic was way back in the thread https://vborg.vbsupport.ru/attachment.php?attachmentid=136597&d=1329854992

[waterart]

thanks i saw the pic and i expanded the entire menu on the left search for ban or spider and it doesnt show up anywhere?

then i installed spam hammer which does show a menu item and shows a link for options and I ended up finding it there.

thanks

[Simon Lloyd]

it would have been found in vbulletin options, if you've added spam hammer uninstall my mod as spam hammer has added pretty much what this mod does to its code, you don't need both.

[SEAda]

Had a legitimate user blocked, removed mod and user still gets redirected.

[Simon Lloyd]

If you've removed the mod then its not this mods fault!, it doesn't keep any database....etc on which UA strings are blocked, if it blocked a legitimate user it's down to what and how you entered the string to ban.

EDIT: You've never downloaded this mod!!!!!

[SEAda]

If you've removed the mod then its not this mods fault!, it doesn't keep any database....etc on which UA strings are blocked, if it blocked a legitimate user it's down to what and how you entered the string to ban.

EDIT: You've never downloaded this mod!!!!!

Yes I did and I have a user who was blocked after installing this mod, and was redirected to the website that is in there by default.

Was hoping for help solving the issue.

[SEAda]

Just because someone is asking for help with your Mod does not mean that that are accusing you of wrong doing. Chill.

[Simon Lloyd]

The way this system here at .org works it tells me if you have downloaded the mod and im afraid it doesn't show downloaded against your name.

To be clear, this mod does not have any residual elements, remove it and everything to do with the mod is removed, there isn't any maintained list or database entries, if the user is still being blocked it's something else on your forum like the cache timeout or indeed their caching.

This mod will only block whatever you enter as a string to look for in the UA, post me the list as you had it so I can help, saying it blocked someone is like me telling you I bought a car and it doesn't work, any help?

And i'm always chilled :D

[SEAda]

The way this system here at .org works it tells me if you have downloaded the mod and im afraid it doesn't show downloaded against your name.

To be clear, this mod does not have any residual elements, remove it and everything to do with the mod is removed, there isn't any maintained list or database entries, if the user is still being blocked it's something else on your forum like the cache timeout or indeed their caching.

This mod will only block whatever you enter as a string to look for in the UA, post me the list as you had it so I can help, saying it blocked someone is like me telling you I bought a car and it doesn't work, any help?

And i'm always chilled :D

Thank you for your help and I apologise if it seemed as if I was disparaging your mod in anyway. It will get a high rating from me.

[SEAda]

I cleared system cache in vBulletin and it resolved my issue. Thanks!

[Simon Lloyd]

You should still post your list so I can give you help on things you may have in it that could cause you further issues.

[Gadget_Guy]

Hi,

I am ditching another spam management product in favour for the ones being suggested by Ozzymods.

This happens to be one piece of the puzzle.


Question....

What should I be doing for the options:

Redirect to spiders own IP?

and

Redirect URL?


Do I say "Yes" to the redirect?

What is the best thing to put in the redirect URL box?

It says this now: www.klikhierniet.net which is the default, but not sure what good this url does or if it should be something else.


D.

[Simon Lloyd]

You can either redirect to a url or back to their own IP its a matter of choice, selecting YES for back to their ip is favourite :)

The default url translates to "Don't click here", you can click it and see what they are faced with.

[Simon Lloyd]

Updated product uploaded.

[Simon Lloyd]

If anyone downloaded the new version before I posted this message please download it again and overwrite the previous one, the first update had a bug :)

[hotshot]

Thanks for the heads up!

[Doug Nelson]

Is this the bug that is keeping me from logging in since I installed it?

[ozzy47]

Ok lets do this, disable plugins via the includes/config.php file.

To do that open your includes/config.php file and below <?php add the following.

define('DISABLE_HOOKS', true);

So it looks like this:
<?php
define('DISABLE_HOOKS', true);
/*================================================= =====================*\
|| ################################################## ################## ||
|| # vBulletin 4.2.2

Use a editor like notepad++ (http://notepad-plus-plus.org/download/v6.6.8.html) to edit any files, don't use Notepad or Wordpad.

Then you can import the updated XML, then reenable hooks.

[Doug Nelson]

Thanks.

[ozzy47]

Not a problem. :)

[Max Taxable]

Is this the bug that is keeping me from logging in since I installed it?The only "bug" was, the new version was continuing to count bots even when off. Nothing to do with your login.

[ozzy47]

Yes there was a conflict in the vB3 version with catching templates, that might have caused the described issue. :) But it is fixed now.

[Max Taxable]

Yes there was a conflict in the vB3 version with catching templates, that might have caused the described issue. :) But it is fixed now.Ahh, didn't realize what thread this was!

Imagine though if, the complaint really had to do with use of IE6....

[ozzy47]

If anyone gets errors like this,

Invalid SQL:

UPDATE `bad_bots_blocked` SET ban_useragent = ban_useragent + 1;

Use the attached XML, I had one of my own updates to my own table in the XML, that should not have been in there.

Simon can grab that XML, and update the OP when he comes on.

[Simon Lloyd]

If anyone gets errors like this,

Invalid SQL:

UPDATE `bad_bots_blocked` SET ban_useragent = ban_useragent + 1;

Use the attached XML, I had one of my own updates to my own table in the XML, that should not have been in there.

Simon can grab that XML, and update the OP when he comes on.

New XML uploaded as requested, labelled version 3.1.2

[Simon Lloyd]

Is this the bug that is keeping me from logging in since I installed it?If you cant log in it's probably because you have something from your own User Agent in the ban list. use the links i give in the mod description to find out more about this, check the "Genuine Members getting blocked" section :)

[Alex6969]

problem V 3.1.2

Fatal error: Cannot redeclare banned_redirect() (previously declared in /home4/XXXXXX/public_html/includes/class_bootstrap.php(103) : eval()'d code:7) in /home4/XXXXXX/public_html/includes/class_bootstrap.php(996) : eval()'d code on line 7

old version no problem

[Simon Lloyd]

problem V 3.1.2

Fatal error: Cannot redeclare banned_redirect() (previously declared in /home4/XXXXXX/public_html/includes/class_bootstrap.php(103) : eval()'d code:7) in /home4/XXXXXX/public_html/includes/class_bootstrap.php(996) : eval()'d code on line 7





old version no problemThats the VB4 version? did you download it from this thread?

[Simon Lloyd]

problem V 3.1.2

Fatal error: Cannot redeclare banned_redirect() (previously declared in /home4/XXXXXX/public_html/includes/class_bootstrap.php(103) : eval()'d code:7) in /home4/XXXXXX/public_html/includes/class_bootstrap.php(996) : eval()'d code on line 7





old version no problemResolved, new attachment to mod description is the correct version :)

[adwade]

Today, 01:58 New XML uploaded as requested, labelled version 3.1.2

ha, ha Of course I downloaded it at 01:32 today :eek:, so seeing this I just re-downloaded and re-imported. Thanxx for the updates! :cool:

[Simon Lloyd]

You imported with overwrite right?, hopefully all should be good - thanks goes to Ozzy47 for the additional functions like stats...etc

[ozzy47]

Nah, the thanks goes to you, for bringing the mod to the community in the first place. :)

[adwade]

You imported with overwrite right?

Yes, that is correct! :up:

[ozzy47]

Yes, that is correct! :up:

Then you now should be fine. :)

[Alex6969]

<font color="Red">All ok.....problem solved.......thanks SIMON !!!</font>

[X-or]

nice mod but after updating to vb 4.2.3 I'm getting a php warning :

PHP User Warning: fetch_template() calls should be replaced by the vB_Template class. Template name: bs_blocked in ..../includes/functions.php on line 4602

someone could fix this please? this is one very useful mod