vB Bad Behavior

[Eric]

/**
* vB Bad Behavior is free software; you can redistribute it and/or modify it under
* the terms of the GNU Lesser General Public License as published by the Free
* Software Foundation; either version 3 of the License, or (at your option) any
* later version.
*
* This program is distributed in the hope that it will be useful, but WITHOUT ANY
* WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
* PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.
*/

What is vB Bad Behavior?
This is an integration of the Bad Behavior software with vBulletin.

What is Bad Behavior?
Bad Behavior is a PHP-based solution for blocking link spam and the robots which deliver it. Bad Behavior complements other link spam solutions by acting as a gatekeeper, preventing spammers from ever delivering their junk, and in many cases, from ever reading your site in the first place. This keeps your site's load down, makes your site logs cleaner, and can help prevent denial of service conditions caused by spammers.

Visit http://bad-behavior.ioerror.us/ for more.

Features
For more information on the features of Bad Behavior (and subsequently this mod) please go to Bad Behavior's site:

http://bad-behavior.ioerror.us/documentation/benefits/

For features related to the mod itself, please take a look at the screenshots.

This mod should work with the entire 3.x series (well, beginning with 3.5), but it's only been tested on 3.8.x. I'm not sure if this works on vB 4.x yet, as I've not tested it - but if you try it out, let me know!

Installation
1. Extract the contents of the zip file.
2. Upload the contents of the `upload` folder to your forum root.
3. Enter your AdminCP and go to Plugins & Products > Manage Products > [Add/Import Product]
4. Import the product using the `product-vb_badbehavior.xml` file.
5. Configure the mod in AdminCP -> vBulletin Options -> vBulletin Options -> vB Bad Behavior Options

Upgrading

vB Bad Behavior
In many cases, all you'll need to do to upgrade is follow the installation instructions above.

The only difference, will be you'll need to allow the files to overwrite. Also, when re-importing the product file, you'll need to set "Allow Overwrite" to "Yes".

Bad Behavior
Bad Behavior's files are at `/includes/bad-behavior/`. If you wish to update manually go to:

http://bad-behavior.ioerror.us/download/

And download the latest development version. Extract the zip, and upload the contents of `bad-behavior` to `/includes/bad-behavior/` allowing the files to overwrite.

Versions
The current version of Bad Behavior this mod is using is: v2.2.14
The current version of Bad Behavior (development) is: v2.2.14

Changelog
Version 1.0.13, 04/23/2013

Bad Behavior upgraded to 2.2.14


Version 1.0.12, 12/21/2012 -- Released: 02/05/2013

Bad Behavior upgraded to 2.2.13
Added some more ranges to whitelist.ini


Version 1.0.10, 09/09/2012

Bad Behavior upgraded to 2.2.10


Version 1.0.9, 06/17/2012

Bad Behavior upgraded to 2.2.7


Version 1.0.8, 06/12/2012

Bad Behavior upgraded to 2.2.6
New Setting: EU Cookie


Version 1.0.7, 05/04/2012

Bad Behavior upgraded to 2.2.3
Cron/Scheduled Task for automatic log pruning added.


Version 1.0.6, 01/04/2012

Bad Behavior upgraded to 2.1.15


Version 1.0.5, 05/26/2011

Added option for bypassing users/members.
If the visitor is a user, and is in usergroup 5, 6, or 7 (admin/mod/super mod) - Bad Behavior is bypassed.
Modified bad-behavior core to check for Google Web Preview

file edited: /includes/bad-behavior/core.inc.php

Added a link beside the IP address in the log for WhoIs.


Version 1.0.4, 04/28/2011

Bad Behavior upgraded to 2.1.13 (fixes search engine block issues)
Added Paypal/Paypal IPN IP address to the whitelist.
Added payment gateway file names to the whitelist.


Version 1.0.3, 04/21/2011

Fix #1: Pruning log doesn't work.
Fix #3: POST more than two days after GET (added support for BB's javascript)
Fix #5: Cannot modify header information error (suppressed error in BB's function)
Implemented #6: Filter per key (new admincp option to list keys not to be shown in log)
Implemented #9: Show link to member profile (if userid is found in headers, link to profile)


Version 1.0.2, 04/10/2011

Updated /includes/functions_vb_badbehavior.php to:

disable Reverse Proxy if Reverse Proxy Addresses are empty
distinguish SQL queries using "SET", for example: SET @@session.wait_timeout = 90 - which is used by BB
set "offsite_forms" to false by default, as it's not really needed in vB IMHO, and it can cause problems with certain setups
cleaned up the bb2_read_settings() function and fixed a typo in one of the vbulletin options calls

Updated /includes/whitelist.ini to include the following GOOGLE ranges:

74.125.0.0/16
216.239.32.0/19
209.85.128.0/17
66.102.0.0/20

Updated /admincp/vb_badbehavior.php

Log pruning was pruning all logs, despite what was entered for number of days



Version 1.0.1, 04/06/2011

Bad Behavior upgraded to 2.1.12
Changed files:

/includes/bad-behavior/core.inc.php
/includes/bad-behavior/searchengine.inc.php

"Verbose" admin option now set to "No" by default.


Version 1.0.0, 04/05/2011

Initial release.



Screenshots
Screenshots can now be seen at: http://www.secondversion.com/images/vb/vb_badbehavior/

I was running out of room for attachments here on vB.org


Development

https://github.com/ericsizemore/vb_bad_behavior/tree/master/vb3


Only those who "Mark As Installed" will receive support for this modification.

[Eric]

Reserved for future use.

[error10]

The screenshots look very nice; I'm going to throw this up on a dev box with 4.1 later and see what happens.

By the way, 2.1.11 is quite stable now; it's going to end up being the 2.2 release candidate shortly (I have to fix some stuff in the MediaWiki port first). Then I can get down to the business of squashing some completely new spammers who have been annoying me lately.

[Alfa1]

Thanks Eric. Does the log show only the users / bots that have been blocked? I see valid users in the log. I have just installed this addon a few minutes ago and already have 491 entries.

[error10]

Thanks Eric. Does the log show only the users / bots that have been blocked? I see valid users in the log. I have just installed this addon a few minutes ago and already have 491 entries.

Hmm. It SHOULD log everything only when verbose mode is enabled in the settings. If it's off, only blocked requests should be logged. Did you enable verbose mode? If so, try disabling it.

[Eric]

If "Verbose" is enabled, it logs everything, but if the code is 00000000, that means they have not been blocked. You may want to set "Verbose" to "No" to only log suspicious requests/bots (including blocks).

EDIT: error10 beat me to it ;) It appears I have "Verbose" defaulted to "Yes", maybe I should change that in the next release.

[error10]

EDIT: error10 beat me to it ;) It appears I have "Verbose" defaulted to "Yes", maybe I should change that in the next release.

I default Verbose to No, since most people don't need it. It's only useful if you're gathering data on spammers that Bad Behavior doesn't yet block.

Speaking of which, I just posted my official announcement (http://bad-behavior.ioerror.us/2011/04/06/bad-behavior-for-vbulletin/). I hope you like it. :)

[Alfa1]

Verbose logging is off. The code is 00000000 for valid users.

As a side note: The Project Honeypot website is not very structured. It took me a while to find where I needed to go here for a BL API key. So here is the url so that others don't have to search: http://www.projecthoneypot.org/httpbl_configure.php
Could be handy to add to the instructions.

[Eric]

I default Verbose to No, since most people don't need it. It's only useful if you're gathering data on spammers that Bad Behavior doesn't yet block.

Speaking of which, I just posted my official announcement (http://bad-behavior.ioerror.us/2011/04/06/bad-behavior-for-vbulletin/). I hope you like it. :)

Yeah, I'll default it to "No" or "Off" in the next release.

And, awesome! Thanks for the blog post, and well.. Bad Behavior itself ;) I have this modification implemented on a forum with nearly 200,000 members, and millions of pageviews/mo - and it's working flawlessly. After 24 hours of use: "There are 3,441 total log entries." (that's not counting the 00000000 entries) :)

[Eric]

Verbose logging is off. The code is 00000000 for valid users.

As a side note: The Project Honeypot website is not very structured. It took me a while to find where I needed to go here for a BL API key. So here is the url so that others don't have to search: http://www.projecthoneypot.org/httpbl_configure.php
Could be handy to add to the instructions.

If those requests are still being logged, it could be the type of request the user is making, but that code indicates they are not being blocked.

EDIT: Thank you for the more direct link for http:BL - I'll be sure to include that. :)

[error10]

If you defaulted it to Yes, but it did not show up as Yes in the Admin CP, then it might just need to be toggled on and back off.

[Alfa1]

I have just toggled the it, and saved it. But it still shows the 00000000 entries. Its no biggie though. I'm happy as can be that Bad Behavior is finally available for vbulletin.

It will be interesting to see how much server resources / bandwidth this will safe for my big board. And how much the spam bot registrations will go down.

[error10]

Sorry to do this to you Eric, but I just pushed an important update (http://bad-behavior.ioerror.us/2011/04/06/bad-behavior-2-0-43-and-2-1-12/) that you'll want to put out as soon as possible.

[viper357]

Yes, thanks for the work on this guys. :)

[Lee G]

Nominated for mod of the month

I love using the earlier version of Bad Behaviour.
This version has just made my day on fighting scrapers and spam bots

Thanks very much for releasing this

[Alfa1]

Does BB submit to ProjectHoneypot? If a bad bot hits my site, then will it be submitted to the database? Or do we need to install additional software to join the honeypot network?

Nominated.

The bots artabus and deepnet explorer are still registering accounts. Can I submit these bots somewhere? Or can I block them with BB?

[Lee G]

If you go into the bad behaviour folder, look for the blacklist.inc.php file
Open it with wordpad or your prefered editor
There are two sections there for banning user agents, first part is begins with user agents
Second part is, if found anywhere user agents

Ban anywhere user agents are after this code
// These user agent strings occur anywhere within the line.
$bb2_spambots = array(

Add the following to the found anywhere section


"artabus",
"Deepnet Explorer",


This mod is so easy to work with on banning or blocking bad user agents
I found this mod cured a problem I have with banning user agents via htaccess

To test if the ban works. Go to Bots vrs Browsers and check on this page
http://www.botsvsbrowsers.com/SimulateUserAgent.asp

Add the user agents anywhere in the test string and try to view your forum

[Eric]

Sorry to do this to you Eric, but I just pushed an important update (http://bad-behavior.ioerror.us/2011/04/06/bad-behavior-2-0-43-and-2-1-12/) that you'll want to put out as soon as possible.
Ah, no probs. I just got home from work, and will be heading to dinner - but I'll update the mod in the next couple hours. :)

Nominated for mod of the month

I love using the earlier version of Bad Behaviour.
This version has just made my day on fighting scrapers and spam bots

Thanks very much for releasing this
Thanks for the nomination Lee :)

Does BB submit to ProjectHoneypot? If a bad bot hits my site, then will it be submitted to the database? Or do we need to install additional software to join the honeypot network?

Nominated.


The bots artabus and deepnet explorer are still registering accounts. Can I submit these bots somewhere? Or can I block them with BB?
Thanks for the nomination as well Alfa1 :) As for ProjectHoneyBot - as far as I'm aware, it does not submit to the project, only checks against the black list - but I'll see what I can do about that.

And thanks to Lee for posting the code for adding those bots, that should work for you.

[Eric]

Version 1.0.1, 04/06/2011
- Bad Behavior upgraded to 2.1.12
- Changed files:
o /includes/bad-behavior/core.inc.php
o /includes/bad-behavior/searchengine.inc.php
- "Verbose" admin option now set to "No" by default.

I've reached the max. number of attachments for this thread - so I'll have to move the screenshots elsewhere. I will do this soon.

[Alfa1]

There seems to be a problem. After running this for 18 hours I have about 8.000 log entries. I do have an active site. A quick glance shows at least 50 valid members were repeatedly blocked from logging in. So there must be many more guests that have been blocked. About 98% of my users are not logged in.
There were 30% less posts today than normal. This could be a coincidence, but we do not normally get this much deviation.

Im getting the first support tickets in. Im not sure if it is related, but I do have vbadvanced installed. One member mentioned that he was able to log in from forum home, but not from the vbadvanced portal.

I have temporarily turned BB off until this issue is resolved. Please let me know if you need any information or if I can do anything to fix it.

[Eric]

There seems to be a problem. After running this for 18 hours I have about 8.000 log entries. I do have an active site. A quick glance shows at least 50 valid members were repeatedly blocked from logging in. So there must be many more guests that have been blocked. About 98% of my users are not logged in.
There were 30% less posts today than normal. This could be a coincidence, but we do not normally get this much deviation.

Im getting the first support tickets in. Im not sure if it is related, but I do have vbadvanced installed. One member mentioned that he was able to log in from forum home, but not from the vbadvanced portal.

I have temporarily turned BB off until this issue is resolved. Please let me know if you need any information or if I can do anything to fix it.
In the logs, you're able to click on the "Key" to see the reason for them being blocked. So if you could let me know what it's showing as the reason, and the URL(s) they were blocked on, I'll take a look. As for vBAdvanced, it's not a product I use personally, but what version are you using?

[Alfa1]

About 50 members causing there instances:
HTTP Response: 403
Explanation: You do not have permission to access this server. Data may not be posted from offsite forms.
Log Message: Referer did not point to a form on this site/forum/login.php?do=login
My website has various urls and runs both http as https.

1 member causing these instances:
HTTP Response: 403
Explanation: You do not have permission to access this server. This may be caused by a malfunctioning proxy server or browser privacy software.
Log Message: A User-Agent is required but none was provided.These are donation / subscription payments that are blocked.
The url for this one relates to my payment module and contains variables relating to payment information.

I did not find logs relating to /index.php or other vbadvanced pages, except for malicious bots hitting it.
Im running vbadvanced 3.2

[Eric]

About 50 members causing there instances:

HTTP Response: 403
Explanation: You do not have permission to access this server. Data may not be posted from offsite forms.
Log Message: Referer did not point to a form on this site

/forum/login.php?do=login

1 member causing these instances:

HTTP Response: 403
Explanation: You do not have permission to access this server. This may be caused by a malfunctioning proxy server or browser privacy software.
Log Message: A User-Agent is required but none was provided.

These are donation / subscription payments that are blocked.
The url for this one relates to my payment module and contains variables relating to payment information.

I did not find logs relating to /index.php or other vbadvanced pages, except for malicious bots hitting it.
Im running vbadvanced 3.2


So for the first one... vBAdvanced posting from / to /forum/login.php seems to be triggering BB as being an "offsite form". Let me look into that.

On the second one, would you mind showing me the header information? (If you'd PM it to me, and remove any type of private information).

[Eric]

Ok, on the first one, given vB's form tokens, you should be able to edit /includes/functions_vb_badbehavior.php and set "offsite_forms" to true in the $bb2_settings_defaults array.

[Alfa1]

Sent by PM.

[Hornstar]

I will watch this one, however I too don't want legit users blocked, so I will see how you go first alfa

[Eric]

After discussing the issue via PM with Alfa1, I have a few ideas for some changes that I'll implement in the next release. For now, if you use this mod, especially if using vBAdvanced or other portals that will allow a user to login from it, edit /includes/vb_badbehavior.php and set 'offsite_forms' to true in the $bb2_settings_defaults array.

I'm going to work right now, so I'll post more once I get home.

[Eric]

Screenshots can now be seen at: http://www.secondversion.com/images/vb_badbehavior/

I was running out of room for attachments here on vB.org - Also, to lower the size of the download... removed the screenshots from the zip files.

[thenetbox]

Things seem to work so far :) Thanks!

When I enter the http:BL API Key, the entire forum goes blank though. White screen, no error.

[thenetbox]

Is there a way to whitelist a few IP addresses or user agents?

Thanks again.

[Eric]

Things seem to work so far :) Thanks!

When I enter the http:BL API Key, the entire forum goes blank though. White screen, no error.
Odd. I've been testing the http:BL for some time, works fine on my end. Any errors in your error log?

Is there a way to whitelist a few IP addresses or user agents?

Thanks again.
Edit /includes/whitelist.ini

[Lee G]

Gone for the install today and all went very easy

Looking through my logs, google and bing seem to get stopped a lot
Im going to white list these in the top level whitelist and see if it cures the problem

[Lee G]

It looks like there is also a complete google range missing 74.125.0.0/16

This is the message I get on most Google bot hits

f1182195

HTTP Response: 403
Explanation: An invalid request was received. You claimed to be a major search engine, but you do not appear to actually be a major search engine.
Log Message: User-Agent claimed to be Googlebot, claim appears to be false.

[Lee G]

A bit more playing around and it looks like google gets blocked when reverse proxy is enabled

[thenetbox]

Odd. I've been testing the http:BL for some time, works fine on my end. Any errors in your error log?


No. There are no errors in the errorlog when the white screen happens. Removing the API key makes everything go back to normal again.

[Eric]

No. There are no errors in the errorlog when the white screen happens. Removing the API key makes everything go back to normal again.
What PHP version are you using?

It looks like there is also a complete google range missing 74.125.0.0/16

This is the message I get on most Google bot hits

f1182195

HTTP Response: 403
Explanation: An invalid request was received. You claimed to be a major search engine, but you do not appear to actually be a major search engine.
Log Message: User-Agent claimed to be Googlebot, claim appears to be false.
Hmm, odd. This appears to be with the Bad Behavior core - you can either add that range to the whitelist.ini file, or edit /includes/bad-behavior/searchengines.inc.php

I'll add this in the next release.

[thenetbox]

What PHP version are you using?


Thanks for following up :)

Using PHP Version 5.2.5

[Lee G]

I have been through and found what option kills the google connections
When you put a tick in the "Reverse Proxy" and leave everything below it as is on install, it blocks Google with the f1182195 error and from what I could see also bing

That still occurred with all the bots ip ranges white listed

Once I found that, it has been working well

[Eric]

Version 1.0.2, 04/10/2011

Updated /includes/functions_vb_badbehavior.php to:

disable Reverse Proxy if Reverse Proxy Addresses are empty
distinguish SQL queries using "SET", for example: SET @@session.wait_timeout = 90 - which is used by BB
set "offsite_forms" to false by default, as it's not really needed in vB IMHO, and it can cause problems with certain setups
cleaned up the bb2_read_settings() function and fixed a typo in one of the vbulletin options calls

Updated /includes/whitelist.ini to include the following GOOGLE ranges:

74.125.0.0/16
216.239.32.0/19
209.85.128.0/17
66.102.0.0/20

Updated /admincp/vb_badbehavior.php

Log pruning was pruning all logs, despite what was entered for number of days

[Lee G]

Nice smooth upgrade
I cant believe how much junk this stops without adding any extra user agents
Just over 3500 log entries since I have been running it

Thanks for all the work your putting into this Eric

[Alfa1]

I think the explanation of what BB is should include that BB also blocks a large number of content scrapers and malicious bots. This saves bandwidth costs and increases security.

I find these very important aspects of BB and adding this will increase the number of sites that install BB.

[thomas]

Thanks for this great mod, Eric!

Version 1.0.2, 04/10/2011 Updated /includes/whitelist.ini to include the following GOOGLE ranges:

74.125.0.0/16
216.239.32.0/19
209.85.128.0/17
66.102.0.0/20


Does the whitelist also include Google's MediaBot (for AdSense)?

[Alfa1]

I have enabled Bad Behavior again. It immediately freed up my server from an insane server load. Server load went from 38 to 0.7 almost instantly. :)

I do see a valid members blocked. Details:

A very large number of these:
Key: HTTP Response: 403
Explanation: You do not have permission to access this server. Before trying again, close your browser, run anti-virus and anti-spyware software and remove any viruses and spyware from your computer.
Log Message: POST more than two days after GET
User agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/4.0
URI: /forum/ajax.php
Entity: security token present.
Headers: POST /forum/ajax.php HTTP/1.1
Host: www.my-forum.com (http://www.my-forum.com)
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/4.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
X-Requested-With: XMLHttpRequest
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Referer: http://www.my-forum.com/forum/forumdisplay.php?f=398
Content-Length: 82
Cookie: bb2_screener_= [omitted by Alfa1]
DNT: 1
Pragma: no-cache
Cache-Control: no-cacheI dont understand how it is possible that a large number of valid user post more than 2 days after GET.

A large number of these:
Key: HTTP Response: 403
Explanation: An invalid request was received from your browser. This may be caused by a malfunctioning proxy server or browser privacy software.
Log Message: Required header 'Accept' missing

UserAgent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16
URI: /forum/misc.php?do=page&template=ncode_opensearch
Entity:
Headers: GET /forum/misc.php?do=page&template=ncode_opensearch HTTP/1.1
Host: www.my-forum.com (http://www.my-forum.com)
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: bb2_screener_= [omitted by Alfa1I find this one worrisome because its in the 2b021b1f key.

Key: HTTP Response: 403
Explanation: You do not have permission to access this server. Before trying again, run anti-virus and anti-spyware software and remove any viruses and spyware from your computer.
Log Message: IP address found on http:BL blacklist
UserAgent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3
URI: /forum/ajax.php
Entity: securitytoken: xxxxxxxxxxxxxxxx
do: securitytoken
ajax: 1
Headers:POST /forum/ajax.php HTTP/1.1
Host: www.my-forum.com (http://www.my-forum.com)
Content-Length: 82
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
X-Requested-With: XMLHttpRequest
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Referer: http://www.my-forum.com/forum/search.php?searchid=2679481
Cookie: bb2_screener_=xxxxxxxxxxxxx
Pragma: no-cache
Connection: keep-alive
I see these valid members are using proxies like TOR and similar.


Key:
UserAgent:
URI:
Entity:
Headers:Feature request 1: for the log: filter per key, so that it is possible to see all entries except those with key 00000 and key 2b021b1f. Or just view all entries with a certain key. That makes it much easier to see the similarities of the entries with the same key.

Feature request 2: Alert the admin which members have been blocked by BB and why. This makes it easier to detect problems with BB and forum accounts registered by bots. I think the optimal way to notify the admin is by PM.

Feature request 3: Trace IP directly from the log.

Feature request 4: related to FR 2. If bbuserid is present in headers then show link to user profile in the log. This makes it easy to check if the blocked members was a valid user or not.

[Alfa1]

Running in debug mode and checking out the queries exposes this error on forum home:
Warning: Cannot modify header information - headers already sent by (output started at /private_html/forum/global.php(355) : eval()'d code:166) in /private_html/forum/includes/bad-behavior/screener.inc.php on line 8
End call of global.php: 0.19540810585

[Lee G]

Just been through my last 450 denies and it looks like a Yahoo bot got the cold shoulder

Bot ip 67.195.112.41

Full ip range 67.195.0.0/16
http://whois.domaintools.com/67.195.112.41

Apart from that, its been working like a dream

[Alfa1]

Its really amazing to see how many attacks, malicious bots and content scrapers this mod is preventing. No wonder my server was under such heavy strain. While it was at crawling pace before, its lightning fast now.

[Eric]

I have enabled Bad Behavior again. It immediately freed up my server from an insane server load. Server load went from 38 to 0.7 almost instantly. :)

I do see a valid members blocked. Details:

A very large number of these:
I dont understand how it is possible that a large number of valid user post more than 2 days after GET.

A large number of these:
I find this one worrisome because its in the 2b021b1f key.
I see these valid members are using proxies like TOR and similar.

Feature request 1: for the log: filter per key, so that it is possible to see all entries except those with key 00000 and key 2b021b1f. Or just view all entries with a certain key. That makes it much easier to see the similarities of the entries with the same key.

Feature request 2: Alert the admin which members have been blocked by BB and why. This makes it easier to detect problems with BB and forum accounts registered by bots. I think the optimal way to notify the admin is by PM.

Feature request 3: Trace IP directly from the log.

Feature request 4: related to FR 2. If bbuserid is present in headers then show link to user profile in the log. This makes it easy to check if the blocked members was a valid user or not.
Regarding: POST more than two days after GET

Looks like this is happening if it's been 48hrs + between the screener cookie and a form submission:

// Posting too fast? 5 sec
// FIXME: even 5 sec is too intrusive
// if ($screener + 5 > time())
// return "408d7e72";
// Posting too slow? 48 hr
if ($screener + 172800 < time())
return "b40c8ddc";

Not sure about that at this point. EDIT: I believe I know why now, will try to address this in the next update.

--

Regarding: Required header 'Accept' missing

The browser/user is sending:

Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3


But not sending "Accept:" for eg:

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8


I'm not sure if that's an issue with Chrome, or what.

--

Regarding: IP address found on http:BL blacklist

Not much to do about that one, as far as this mod IMHO - since that's using data from projecthoneypot.

--

On the feature requests, I'll try to get those in the next release :)

Running in debug mode and checking out the queries exposes this error on forum home:
Hmm. It appears this occurs when BB is trying to set a cookie using the bb2_screener_cookie() function. This could be caused by output from vB/a hook before BB is called.

Just been through my last 450 denies and it looks like a Yahoo bot got the cold shoulder

Bot ip 67.195.112.41

Full ip range 67.195.0.0/16
http://whois.domaintools.com/67.195.112.41

Apart from that, its been working like a dream
Will update the whitelist, ty :) - Actually, BB already checks this range :/

[Eric]

SVN now available at http://subversion.assembla.com/svn/vb-bad-behavior/trunk/vb3/
Trac as well: http://trac.assembla.com/vb-bad-behavior/

[Alfa1]

Log bug added to trac: http://trac.assembla.com/vb-bad-behavior/ticket/1

[Lee G]

If anyone runs the vbseo sitemap generator, check it still works
Mine needs an extra kick every day now since running this mod
I have to go into the sitemap control panel and do a manual generation to finish mine
With this mod disabled, no problems

Apart from that, this is still dropping idiots like there is no tomorrow

[Alfa1]

Does BB block XRumer (http://en.wikipedia.org/wiki/XRumer)? Some forums seem to be getting hacked by this.

I didn't check vbSEO sitemap yet, but since almost everyone uses it, this would be important to resolve.

I entered bugs and feature requests into trac: http://trac.assembla.com/vb-bad-behavior/report/1
I see you already have a spammer in the bug tracker.

[Lee G]

Finally got everything up and running smooth again over night and its worth it
My sitemap is back. The guys over at vbseo have been helping with tips on what server resources to increase

Added a few corporate spiders to the block list to free up some resources along with killing a lot of unused garbage.

The corporate spiders are the ones that do nothing else than search your forums for companies. No online public access for any of their results. The likes of Spin3r etc

Im already starting to see genuine results from using this

[Eric]

Finally got everything up and running smooth again over night and its worth it
My sitemap is back. The guys over at vbseo have been helping with tips on what server resources to increase

Added a few corporate spiders to the block list to free up some resources along with killing a lot of unused garbage.

The corporate spiders are the ones that do nothing else than search your forums for companies. No online public access for any of their results. The likes of Spin3r etc

Im already starting to see genuine results from using this
Was it a problem with BB for the sitemap, or did it turn out to be something else?

Does BB block XRumer (http://en.wikipedia.org/wiki/XRumer)? Some forums seem to be getting hacked by this.

I didn't check vbSEO sitemap yet, but since almost everyone uses it, this would be important to resolve.

I entered bugs and feature requests into trac: http://trac.assembla.com/vb-bad-behavior/report/1
I see you already have a spammer in the bug tracker.
For XRumer... honestly, I'm not sure. And thanks for everything you've logged in Trac - I've been able to close some of them as fixed.

[Simon Lloyd]

I added this today, seems greate because i want to get rid of the stuff from my .htaccess, anyway, it may be that i don't understand it properly yet but the few already blocked seem to come from google although im probably wrong:
Code: 69920ee5
GET /forumz/showthread.php?t=206014 HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: no
Connection: Keep-Alive
Cookie: vbet_language=no; tccsessionhash=1cf31725ad89f8567e17c690468d1efa; tcclastvisit=1302894970; tcclastactivity=0; vbet_sessionUsed=1; __utmc=118899148; __utma=118899148.1559634400.1302894986.1302894986. 1302894986.1; __utmb=118899148.1.10.1302894986; __utmz=118899148.1302894986.1.1.utmcsr=google|utmc cn=(organic)|utmcmd=organic|utmctr=convertere%20fr a%20excel%20til%20txt%20fil%20vba; __utmv=118899148.usergroup-1-Unregistered%20%2F%20Not%20Logged%20In
Host: www.thecodecage.com
Referer:
User-Agent: Mozilla/4.0 (compatible; MSIE 999.1; Unknown)
X-Rewrite-Url: /forumz/showthread.php?t=206014
It seems to show and organic view from google being blocked, i am using the API key but not reverse proxy or verbose.

This is their host: 21.79-160-180.customer.lyse.net
and this is their IP: 79.160.180.21

I use vbenterprisetranslator (paid version) if that helps.

Any ideas?

[Lee G]

Thanks for this mod Eric.
It not only saved my forum but also the contents of a sub folder when some nice Russian person decided to attack my server today.

They were hitting me with all forms of their favorite toy.
libwww-perl/5.811
libwww-perl/5.812
libwww-perl/5.803
libwww-perl/5.837

The logging of blocked ip is brilliant for tracking that kind of hit down

Im now over the 16k logged ips in the short time running this

The vbseo sitemap. I just set it to run when Im up, so if it stops, I can complete the daily update. Without the mod running, the sitemap runs fine. If it runs when the low life that inhabit the internet are on a porn break, it runs fine

[Simon Lloyd]

Maybe this thread explains why some organics are being blocked http://answers.microsoft.com/en-us/windows/forum/windows_other-windows_update/microsoft-update-also-thinks-my-pc-is-a-mac/337affa0-d280-4cdc-bd71-4371c27ca816

EDIT: i just googled Mozilla/4.0 (compatible; MSIE 999.1; Unknown)

Test driving the useragent at http://www.botsvsbrowsers.com/SimulateUserAgent.asp?UserAgent=Mozilla%2F4%2E0+%2 8compatible%3B+MSIE+999%2E1%3B+Unknown%29 shows:
Method: GET
Request Status: 200 : OK
Content-Length: bytes (~k)
Content-Type: text/html; charset=ISO-8859-1
Server: WebServerX

[Eric]

Anyone want to test out 1.0.3 before I release it tomorrow?

http://trac.assembla.com/vb-bad-behavior/changeset/23/tags/1.0.3?old_path=%2F&format=zip

Current changelog...

- Fix #1: Pruning log doesn't work.
- Fix #3: POST more than two days after GET (added support for BB's javascript)
- Fix #5: Cannot modify header information error (suppressed error in BB's function)
- Implemented #6: Filter per key (new admincp option to list keys not to be shown in log)
- Implemented #9: Show link to member profile (if userid is found in headers, link to profile)
- Changes: http://trac.assembla.com/vb-bad-behavior/changeset?new=17%40trunk%2Fvb3&old=6%40trunk%2Fvb3

[Alfa1]

Regarding the Release Candidate: When viewing the log (after pruning) I get this error:
Fatal error: Call to undefined function bb2_log_userid() in /private_html/forum/admincp/vb_badbehavior.php on line 196
Uploading vb_badbehavior.php version 1.0.2 resolves this error.

Fix #3 does not work. Im still getting that issue.

I cant test this for long, because I need issue #2 & #4 fixed. #4 is causing many false negatives.

An interesting note: my big board just had a DDoS attack and server load went trough the roof (Somewhere over 45!!!). Turning on Bad Behavior quickly resolved the situation.

[Eric]

I know why #1 is happening - #3 may need some time to work itself out, since it's based on cookies as well. Not much I can do about #4 - #2 you may just need to whitelist the php script - you can enter the URL into ./includes/whitelist.ini

[Eric]

Version 1.0.3, 04/21/2011

Fix #1: Pruning log doesn't work.
Fix #3: POST more than two days after GET (added support for BB's javascript)
Fix #5: Cannot modify header information error (suppressed error in BB's function)
Implemented #6: Filter per key (new admincp option to list keys not to be shown in log)
Implemented #9: Show link to member profile (if userid is found in headers, link to profile)
Changes: http://trac.assembla.com/vb-bad-behavior/changeset?new=29%40trunk%2Fvb3&old=6%40trunk%2Fvb3

[Alfa1]

Thanks for the update!
#3 may need some time to work itself out, since it's based on cookies as well.
My memberbase is 90.000 large. Some of these may not visit again until years from now and cookies may remain on a computer for a few years, so this would take a very long time to fade out. If it would fade out.
Not much I can do about #4
Could you explain what the issue is? Do you have any idea why so many valid users have 'accept' missing in the header? Are they using some kind of inprivate browsing?

Could you add the key filter to the top of the log view page, so that the log can be filtered while browsing the log?

I posted this Article on vbulletin.com:
Bulletproof spam & scraper protection (http://www.vbulletin.com/forum/showthread.php/378134-Bulletproof-spam-amp-scraper-protection)

[Simon Lloyd]

I have had this mod installed now for 5 days, it has prevented many thousands of attempts to access images....etc and in turn has saved me around 80GB in bandwidth already compared to the same period last month -so a great big THANKS to Eric!

just about to upgrade to the latest version :)

[Lee G]

28k and still counting :D
Nice smooth upgrade
Now its time to integrate my old block list with the new version and that number should go up at a fair rate

[Eric]

Thanks for the update!

My memberbase is 90.000 large. Some of these may not visit again until years from now and cookies may remain on a computer for a few years, so this would take a very long time to fade out. If it would fade out.

Could you explain what the issue is? Do you have any idea why so many valid users have 'accept' missing in the header? Are they using some kind of inprivate browsing?

Could you add the key filter to the top of the log view page, so that the log can be filtered while browsing the log?

I posted this Article on vbulletin.com:
Bulletproof spam & scraper protection (http://www.vbulletin.com/forum/showthread.php/378134-Bulletproof-spam-amp-scraper-protection)
Alfa1, I do apologize, but I have no idea right now what the issue is with the Accept header. There are a few possibilities, however, such as if the user is using a proxy/VPN... or if they are running the browser in "private" mode - there is also some PC software that could cause the problem. I'm going to talk with Michael (error10) about this, and see if he has any ideas.

As far as adding the key filter to the log page, should be possible. And great article :)
I have had this mod installed now for 5 days, it has prevented many thousands of attempts to access images....etc and in turn has saved me around 80GB in bandwidth already compared to the same period last month -so a great big THANKS to Eric!

just about to upgrade to the latest version :)
28k and still counting :D
Nice smooth upgrade
Now its time to integrate my old block list with the new version and that number should go up at a fair rate

Nice! :) And no problem, I'm glad this mod is so useful to some of you.

[Simon Lloyd]

Eric, on your new release i am getting some blocked which show UserId: as blank and the exact same IP but a slightly different GET does not display UserId:

Is there a possibilty fix that, they're obviously not users but must be delivering a call that is mimicking a user but there's no confirmed login...etc, i might be talking twaddle as i don't understand the GET and all the garb that follows it but i hope you get the gist :)

[Lee G]

Same problem being experienced as Simon above

[Eric]

Same problem being experienced as Simon above

Eric, on your new release i am getting some blocked which show UserId: as blank and the exact same IP but a slightly different GET does not display UserId:

Is there a possibilty fix that, they're obviously not users but must be delivering a call that is mimicking a user but there's no confirmed login...etc, i might be talking twaddle as i don't understand the GET and all the garb that follows it but i hope you get the gist :)

Hmm. Could you try something for me? Edit `/includes/functions_vb_badbehavior.php` and replace the `bb2_log_userid` function with:

// Determines if vB Bad Behavior has blocked a user request
// Checks to see of userid is present in the Cookie header
function bb2_log_userid($headers)
{
if (!empty($headers))
{
$_tmp = explode("\n", $headers);
$_tmp = implode('', array_filter($_tmp, '__walker'));
$_tmp = str_replace(';', '&', $_tmp);

if (empty($_tmp))
{
return false;
}
parse_str($_tmp);

$userid = COOKIE_PREFIX . 'userid';

return iif($$userid > 0, $$userid, false);
}
return false;
}

[Simon Lloyd]

Ok just done (07:07 GMT) will report back later today as i get many thousands a day getting caught in this :)

EDIT: Just cleared all logs too!

[Simon Lloyd]

Ok here's some results, this one was captured, didn't show UserId and isn't registered or unconfirmed:
HEADER
POST /forumz/login.php?do=login HTTP/1.0
Accept: */*
Cookie: tccsessionhash=e575197d38d5a7c06fe82415e7688d00; tcclastvisit=1303620326; tcclastactivity=0; vbet_sessionUsed=1
Host: www.thecodecage.com
Pragma: no-cache
Proxy-Connection: Keep-Alive
Referer: http://www.thecodecage.com/register.html?agreed=true
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
ENTITY
vb_login_username: elderatot
cookieuser: 1
vb_login_password: 18gcAEp796
submit: Log in
s:
securitytoken: guest
do: login
vb_login_md5password: 8fb830d941xxxxxxxxbfe29fcc8
vb_login_md5password_utf: 8fb830d9412xxxxxxxxbfe29fcc8
ajax: 0
KEY
2b021b1f

Just in summary, i went through 1,000 (no really i did :)) entries and none of them including the one above showed UserId either filled in or blank.

Hope that helps.

[error10]

Hey all,

For some reason I wasn't getting notifications despite being subscribed to the thread. I'm going back through the messages to see what, if anything, needs to be added or changed in the Bad Behavior core. I'll push an update out as soon as I'm done.

[error10]

These are donation / subscription payments that are blocked.
The url for this one relates to my payment module and contains variables relating to payment information.


PayPal has a long history of sending their IPN notifications without a User-Agent. There's nothing I've been able to do to convince them to send a User-Agent except to advise affected people to complain to PayPal. In the meantime you can whitelist their IP addresses.

A bit more playing around and it looks like google gets blocked when reverse proxy is enabled

If this happens, make sure that X-Forwarded-For is actually the header that your load balancer or accelerator is setting when it forwards HTTP requests to your server. If it uses a different header, be sure to change it. You may also need to list the IP address(es) for your load balancer.

If you aren't using a reverse proxy or load balancer, then you should not enable this option.

Regarding: POST more than two days after GET

Looks like this is happening if it's been 48hrs + between the screener cookie and a form submission:


Eric, the cookie needs to be refreshed on every page load, especially for logged-in users.

If caching is in use, then the cached pages need to be expired at least every 48 hours.

Alfa1, I do apologize, but I have no idea right now what the issue is with the Accept header. There are a few possibilities, however, such as if the user is using a proxy/VPN... or if they are running the browser in "private" mode - there is also some PC software that could cause the problem. I'm going to talk with Michael (error10) about this, and see if he has any ideas.


Most of the time, these are actual spambots.

The rest of the time, it's somebody who installed Norton or something and whatever they're using is stripping out random headers, and the user doesn't really know what's going on. Or someone who thinks they know what they're doing who is a bit extreme with their "privacy". Often these require somebody to actually talk with the user and figure out what's actually going on.

Like Eric, I'm glad my code has been helpful in reducing the spam and DoS problems for your forums. I'm nearing the 2.2 core release and as soon as I have that out, I can get back to work on some core stuff that's been waiting a long time. I'll be posting an updated roadmap for 3.0 soon.

[error10]

Eric, I've pushed out 2.1.13 (http://bad-behavior.ioerror.us/2011/04/25/bad-behavior-2-1-13/) which should resolve the Google and Yahoo search engine issues.

[Simon Lloyd]

Error10, how do we use your latest release?
Also, this may sound a bit daft to you but i'm sure there are many other users of your efforts that want to ask, could you explain (because i don't have a clue) what each part of this is and what it does, i, like most forum owners are paranoid at denying real users or visitors, so it would be a great help (or perhaps release it as an article on your site)

Here's a POST onePOST http://www.thecodecage.com/forumz/members-access-database-functions/188003-re-outlook-6-will-not-send-messages-web-site-links.html/register.php HTTP/1.1
Accept-Encoding: gzip, deflate
Accept-Language: ru-RU
Connection: Close
Host: www.thecodecage.com
User-Agent: Mozilla/4.0 (MSIE 6.0; Windows NT 5.1)
Here's a GET oneGET /forumz/showthread.php?t=162877 HTTP/1.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset: ISO-8859-2,utf-8;q=0.7,*;q=0.7
Accept-Encoding: gzip, deflate
Accept-Language: pl,en-us;q=0.7,en;q=0.3
Connection: keep-alive
Cookie: tcclastvisit=1302940743; tcclastactivity=0; tccuserlgv=1; __utma=118899148.698522646.1302940750.1302940750.1 302940750.1; __utmz=118899148.1302940750.1.1.utmcsr=google|utmc cn=(organic)|utmcmd=organic|utmctr=kod%20%20vba%20 has%C5%82o; __utmv=118899148.usergroup-1-Unregistered%20%2F%20Not%20Logged%20In; tccsessionhash=39baa5cc5c25fad88452daba12603a3f; vbet_sessionUsed=1
Host: www.thecodecage.com
Keep-Alive: 115
Referer: http://www.google.pl/search?q=kalendarz+vba+excel&hl=pl&client=firefox-a&rls=org.mozilla:pl:official&prmd=ivns&ei=HGa0TZThD4qAOu3f3akJ&start=20&sa=N
User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
X-Rewrite-Url: /forumz/showthread.php?t=162877
If we know exactly what we are looking at or for it would help, and of course you will have more constructive questions :)

[Alfa1]

Alfa1, I do apologize, but I have no idea right now what the issue is with the Accept header. There are a few possibilities, however, such as if the user is using a proxy/VPN... or if they are running the browser in "private" mode - there is also some PC software that could cause the problem. I'm going to talk with Michael (error10) about this, and see if he has any ideas.
Until there is a solution for 'Required header 'Accept' missing', is there a way to not block users for this reason? Its blocking about 50 valid users every 24 hours. I have no doubt that its caused by registered members with security software. I do not want to block these real users. Talking to all these users or whitelisting all their IPs is not possible.
Most of the time, these are actual spambots.
In the logs of my limited testing these have been 100% real users.
PayPal has a long history of sending their IPN notifications without a User-Agent. There's nothing I've been able to do to convince them to send a User-Agent except to advise affected people to complain to PayPal. In the meantime you can whitelist their IP addresses.
I would be highly surprised if anyone would be able to convince paypal about anything. I have whitelisted the script.

New feature request:
Alert staff if registered member performs SQL injection or other attacks (http://trac.assembla.com/vb-bad-behavior/ticket/10)

One thing that I find missing in this addon is a way to feed bad bot data to the blacklist. Please consider to add such functionality. Either as part of this addon or as Projecthoneypot integration. Added to tracker: Feed data to blacklist (http://trac.assembla.com/vb-bad-behavior/ticket/11)

[Simon Lloyd]

I would be highly surprised if anyone would be able to convince paypal about anything. I have whitelisted the script.Could you give details of this whitelisting as i'd like to do the same :)

[Alfa1]

In /includes/whitelist.ini find: example.php
Replace with the script that you want to whitelist.

[Simon Lloyd]

:), i know how to whitelist i'd like to know what you did to whitelist paypal?

[Alfa1]

:), i know how to whitelist i'd like to know what you did to whitelist paypal?
I whitelisted the php file that handles my subscriptions.

[carsafety]

I whitelisted the php file that handles my subscriptions.

I've been watching this mod, planning to install it soon. Do all of these things like search engine spiders, Paypal, adsense and other legitimate scripts come whitelisted out of the box?

If not, is there a list of them somewhere with simple instructions on how to add them?

[error10]

Error10, how do we use your latest release?
Also, this may sound a bit daft to you but i'm sure there are many other users of your efforts that want to ask, could you explain (because i don't have a clue) what each part of this is and what it does, i, like most forum owners are paranoid at denying real users or visitors, so it would be a great help (or perhaps release it as an article on your site)

Here's a POST oneHere's a GET oneIf we know exactly what we are looking at or for it would help, and of course you will have more constructive questions :)

Simon, for vBulletin you don't necessarily use it directly; but wait for Eric to package it up and post the update. Bad Behavior consists of two parts, the core code which does the work of deciding whether something is bad or not, and a platform connector which lets it talk to vBulletin (or WordPress or MediaWiki or Drupal or whatever). I maintain the core, and Eric maintains the vBulletin connector, packaging the two together into a single downloadable mod. If Eric ever got run over by a bus, it would be possible to take the core and add it in yourself, but let's hope nobody ever gets run over by a bus.

As for the two entries you posted, the Log entry gives an indication of what the issue was, and of course with POST requests you can inspect the entity. The first one is a pretty blatant registration spam. I'm not sure what the issue is with the second one. Perhaps it was on Project Honey Pot? It doesn't look like you provided the log entry for them, so I can't really be certain.

Until there is a solution for 'Required header 'Accept' missing', is there a way to not block users for this reason? Its blocking about 50 valid users every 24 hours. I have no doubt that its caused by registered members with security software. I do not want to block these real users. Talking to all these users or whitelisting all their IPs is not possible.

In the logs of my limited testing these have been 100% real users.

I would be highly surprised if anyone would be able to convince paypal about anything. I have whitelisted the script.

New feature request:
Alert staff if registered member performs SQL injection or other attacks (http://trac.assembla.com/vb-bad-behavior/ticket/10)

One thing that I find missing in this addon is a way to feed bad bot data to the blacklist. Please consider to add such functionality. Either as part of this addon or as Projecthoneypot integration. Added to tracker: Feed data to blacklist (http://trac.assembla.com/vb-bad-behavior/ticket/11)

I don't want to block real users either, if I can avoid it. But see directly above for Simon's posting of a registration spam, where the spammer has omitted the Accept: header. And obviously not everything is foreseeable. Legitimate users caught by this already get a message stating that it's likely caused by their browser privacy software and some basic instructions on reconfiguring the software. These could always be improved if I knew the specific software causing the problem. I could also move this test to strict mode, though since it actually does block a lot of spam, I fear it would make Bad Behavior almost useless. So this is a hard problem.

A way to send in data, both on bad bots and on legitimate users inappropriately blocked, is on my roadmap already. As for notifying the admin of particular events, I think that will be on Eric.

[Eric]

Updated Bad Behavior core to 2.1.13, but it may be a little bit before an official release of the mod, as I plan on making further changes. For the time being:

http://trac.assembla.com/vb-bad-behavior/export/34/trunk/vb3/upload/includes/bad-behavior/searchengine.inc.php
http://trac.assembla.com/vb-bad-behavior/export/34/trunk/vb3/upload/includes/bad-behavior/core.inc.php

You can download those files, then overwrite the corresponding files in: /yourforum/includes/bad-behavior/

[Lee G]

I've been watching this mod, planning to install it soon. Do all of these things like search engine spiders, Paypal, adsense and other legitimate scripts come whitelisted out of the box?

If not, is there a list of them somewhere with simple instructions on how to add them?

Whitelisting is easy to do.
In the download folder there is a file which is self explanatory when you open it with an editor like microsoft word pad.

The file to edit is
Includes > Whitelist

To add bad user agents is easier than editing your htaccess file.

Includes > bad-behavior > blacklist.inc.php

That file when you open again is easy to add bad user agents to.
First half is starts with, second half is anywhere in the user agent

Its very easy to work with
Out the box it works very well
I personally still block a lot of ip via htaccess
What you might find is someone will hit you twenty times or more in some cases with different user agents
Some get caught, some dont. If you look at your bb blocks in the logs, these people tend to stand out so you can add an ip block

Its a great bit of software to use in your fight against low life

[carsafety]

Whitelisting is easy to do.
In the download folder there is a file which is self explanatory when you open it with an editor like microsoft word pad.

The file to edit is
Includes > Whitelist


Thanks!

Mostly, I'm concerned with whitelisting known entities like Google search, Adsense, Yahoo, Bing, Paypal, etc. Are these generally included int he default whitelist file? I'd have no idea what all to add to whitelist in therms of agents or IP. Obviously, I don't want to block search engines or other "friendly" scripts that should have access to my website.

[error10]

Thanks!

Mostly, I'm concerned with whitelisting known entities like Google search, Adsense, Yahoo, Bing, Paypal, etc. Are these generally included int he default whitelist file? I'd have no idea what all to add to whitelist in therms of agents or IP. Obviously, I don't want to block search engines or other "friendly" scripts that should have access to my website.

All of these except PayPal should be in the latest update.

PayPal is an outlier because they refuse to identify themselves in their IPN notifications, so they get blocked by default. The best thing to do with this is to add a URL whitelist entry for your payment gateway URL.

[Eric]

All of these except PayPal should be in the latest update.

PayPal is an outlier because they refuse to identify themselves in their IPN notifications, so they get blocked by default. The best thing to do with this is to add a URL whitelist entry for your payment gateway URL.

Speaking of... updated http://trac.assembla.com/vb-bad-behavior/browser/trunk/vb3/upload/includes/whitelist.ini?rev=36

[Alfa1]

Speaking of... updated http://trac.assembla.com/vb-bad-behavior/browser/trunk/vb3/upload/includes/whitelist.ini?rev=36
Should the /forum/ directory not be defined, so that it becomes /forum/payment_gateway.php

I could also move this test to strict mode, though since it actually does block a lot of spam, I fear it would make Bad Behavior almost useless. So this is a hard problem.
Please consider to make a third mode: 'medium mode' and add this to it.
This mode would be useful for boards that do not want to block valid users, even if it lets some bots through.
I really can not afford to block 80 registered members per day and thats what happening now. Most users just use security software without knowing how to manage it. They are not adjusting their browsing behavior after the notice from BB. Most would not know where to start.

[error10]

Should the /forum/ directory not be defined, so that it becomes /forum/payment_gateway.php

If your payment gateway URL is http://www.example.com/forum/payment_gateway.php then you would put in /forum/payment_gateway.php .

Please consider to make a third mode: 'medium mode' and add this to it.
This mode would be useful for boards that do not want to block valid users, even if it lets some bots through.
I really can not afford to block 80 registered members per day and thats what happening now. Most users just use security software without knowing how to manage it. They are not adjusting their browsing behavior after the notice from BB. Most would not know where to start.

I think I have an idea of how to solve this problem without moving to strict mode. Give me a day or so. :)

[Eric]

If your payment gateway URL is http://www.example.com/forum/payment_gateway.php then you would put in /forum/payment_gateway.php .



I think I have an idea of how to solve this problem without moving to strict mode. Give me a day or so. :)
Were you able to come up with something? :) I'll hold off on the next release if so, that way I can incorporate it first.

[Simon Lloyd]

Ok here's some results, this one was captured, didn't show UserId and isn't registered or unconfirmed:
HEADER
POST /forumz/login.php?do=login HTTP/1.0
Accept: */*
Cookie: tccsessionhash=e575197d38d5a7c06fe82415e7688d00; tcclastvisit=1303620326; tcclastactivity=0; vbet_sessionUsed=1
Host: www.thecodecage.com
Pragma: no-cache
Proxy-Connection: Keep-Alive
Referer: http://www.thecodecage.com/register.html?agreed=true
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
ENTITY
vb_login_username: elderatot
cookieuser: 1
vb_login_password: 18gcAEp796
submit: Log in
s:
securitytoken: guest
do: login
vb_login_md5password: 8fb830d941xxxxxxxxbfe29fcc8
vb_login_md5password_utf: 8fb830d9412xxxxxxxxbfe29fcc8
ajax: 0
KEY
2b021b1f

Just in summary, i went through 1,000 (no really i did :)) entries and none of them including the one above showed UserId either filled in or blank.

Hope that helps.Hi Eric, it seems that the revised code you gave with regards to the UserId is working fine, i reported the above to you after changing the code, i have just has a genuine user caught by the mod and this time it did display UserId just fine :)

[Eric]

Updated...

Version 1.0.4, 04/28/2011
- Bad Behavior upgraded to 2.1.13 (fixes search engine block issues)
- Added Paypal/Paypal IPN IP address to the whitelist.
- Added payment gateway file names to the whitelist.
- Changes: http://trac.assembla.com/vb-bad-behavior/changeset?new=43%40trunk%2Fvb3&old=30%40trunk%2Fvb3

[viper357]

Please consider to make a third mode: 'medium mode' and add this to it.
This mode would be useful for boards that do not want to block valid users, even if it lets some bots through.
I really can not afford to block 80 registered members per day and thats what happening now. Most users just use security software without knowing how to manage it. They are not adjusting their browsing behavior after the notice from BB. Most would not know where to start.

I think I have an idea of how to solve this problem without moving to strict mode. Give me a day or so. :)
Were you able to come up with something? :) I'll hold off on the next release if so, that way I can incorporate it first.
Updated...Does this update include the above fix/mode? If it's blocking 80 registered users per day then that's about half of my active membership :p

[Alfa1]

Deepnet explorer and artabus are still trying to register spam accounts. My blacklist has this:
// These user agent strings occur anywhere within the line.
$bb2_spambots = array(
"\r", // A really dumb bot
"; Widows ", // misc comment/email spam
"a href=", // referrer spam
"Bad Behavior Test", // Add this to your user-agent to test BB
"compatible ; MSIE", // misc comment/email spam
"compatible-", // misc comment/email spam
"DTS Agent", // misc comment/email spam
"Email Extractor", // spam harvester
"Gecko/25", // revisit this in 500 years
"grub-client", // search engine ignores robots.txt
"hanzoweb", // very badly behaved crawler
"Indy Library", // misc comment/email spam
"MSIE 7.0; Windows NT 5.2", // Cyveillance
"Murzillo compatible", // comment spam bot
".NET CLR 1)", // free poker, etc.
"POE-Component-Client", // free poker, etc.
"Turing Machine", // www.anonymizer.com (http://www.anonymizer.com) abuse
"unspecified.mail", // stealth harvesters
"User-agent: ", // spam harvester/splogger
"WebaltBot", // spam harvester
"WISEbot", // spam harvester
"WISEnutbot", // spam harvester
"Windows NT 4.0;)", // wikispam bot
"Windows NT 5.0;)", // wikispam bot
"Windows NT 5.1;)", // wikispam bot
"Windows XP 5", // spam harvester
"WordPress/4.01", // pingback spam
"Xedant Human Emulator",// spammer script engine
"\\\\)", // spam harvester
"artabus",
"Deepnet Explorer",
"DigExt",
"MarketwireBot",
"SoftLayer Server",
"FairShare",
"MRSPUTNIK",
"HackerTarget.com",
"JoBo",
"EMail Exractor",
"radian6",
"Alexa",
"boardpulse",
"harvest",
"Wget",
"HTTrack",
"copy",
"copier",

);What am I doing wrong?
Please add these bots to the default blacklist.



Does this update include the above fix/mode? If it's blocking 80 registered users per day then that's about half of my active membership :p
No, its not resolved yet.
http://trac.assembla.com/vb-bad-behavior/ticket/4

[Eric]

Deepnet explorer and artabus are still trying to register spam accounts. My blacklist has this:
What am I doing wrong?
Please add these bots to the default blacklist.




No, its not resolved yet.
http://trac.assembla.com/vb-bad-behavior/ticket/4
I'll see about adding them by default to the blacklist, maybe I could have error10 add them to BB itself as well.

Does this update include the above fix/mode? If it's blocking 80 registered users per day then that's about half of my active membership :p
Right now, I have not heard back from error10 yet, so no that was not included. Even if installed, you won't necessarily block many (if any) users - just seems to be users that may be using "private" software w/their browser.

[Lee G]

I found another Google range has been getting denied access

User agent
Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13

IP range
72.14.192.0 - 72.14.255.255 or 72.14.192.0/18

Alpha1. If you think a user agent is not being denied, try it over at bits vrs browsers on the user agent test page
http://www.botsvsbrowsers.com/SimulateUserAgent.asp
You might find your artabus block needs a capital A

Fairshare from memory are a limited ip range.
I know I block them via my htaccess. But I cant remember what their ip range is

[Alfa1]

I added 2 important feature requests:

http://trac.assembla.com/vb-bad-behavior/ticket/12
When a registered member is blocked by BB for one of the following reasons, send the member a message by PM which explains the issue and informs the member how to resolve it.
17f4e8c8 User-Agent was found on blacklist
2b021b1f IP address found on http:BL blacklist

This allows valid members to address the issue and switch anonymous proxies, or take other measures to resolve.http://trac.assembla.com/vb-bad-behavior/ticket/13
Please add a admincp setting to turn off blocking of registered members.
This setting can be extremely valuable for admins that encounter problems with registered members being blocked by vb BB.

I can envision two versions of this setting:
1. IF bbuserid is found, then bypass BB
2. IF bbuserid is found AND joindate is older than 60 days, then bypass BBCurrently I see a lot of users blocked by the accept issue and must disable BB because of this. The above function would allow me to keep BB running while the issue is addressed.

IMO this ticket about the accept issue should be reopened, as you are trying to resolve the issue:
http://trac.assembla.com/vb-bad-behavior/ticket/4

[Lee G]

If only this hooked up with the Stop Forum Spam database. It would take out even more of the proxies the scrapers and low life hit from.

Project honey pot only seems to mark ips that have been caught trying to send emails

Im presently running it set on 1 rather than the default 20 or 25 that is set default

None of my members have been blocked. Then again, unless they are trying to avoid bans, they dont use proxies

[Eric]

I found another Google range has been getting denied access

User agent
Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13

IP range
72.14.192.0 - 72.14.255.255 or 72.14.192.0/18

Alpha1. If you think a user agent is not being denied, try it over at bits vrs browsers on the user agent test page
http://www.botsvsbrowsers.com/SimulateUserAgent.asp
You might find your artabus block needs a capital A

Fairshare from memory are a limited ip range.
I know I block them via my htaccess. But I cant remember what their ip range is

That range for Google is already listed - how are you checking if it's denied, or is it in your log?

[Eric]

If only this hooked up with the Stop Forum Spam database. It would take out even more of the proxies the scrapers and low life hit from.

Project honey pot only seems to mark ips that have been caught trying to send emails

Im presently running it set on 1 rather than the default 20 or 25 that is set default

None of my members have been blocked. Then again, unless they are trying to avoid bans, they dont use proxies

I plan on adding stopForumSpam actually. ;)

I added 2 important feature requests:

http://trac.assembla.com/vb-bad-behavior/ticket/12
http://trac.assembla.com/vb-bad-behavior/ticket/13
Currently I see a lot of users blocked by the accept issue and must disable BB because of this. The above function would allow me to keep BB running while the issue is addressed.

IMO this ticket about the accept issue should be reopened, as you are trying to resolve the issue:
http://trac.assembla.com/vb-bad-behavior/ticket/4
The best way to handle bypassing members would be to only run the plugin for guests - which I can code that in.

[Lee G]

Hi Eric
The google range that was being blocked, I found in my logs.
Put the range in the white list and it cured the problem

As for adding the stop forum spam database, you have just made my day

[Simon Lloyd]

I have stopforumspam installed and did use it for a little while but i was getting many many real people getting blocked & it was allowing registrations if they didn't provide an email address, so now i just have it set to allow but log evrything. It needs a bit of extra work to work better but another great tool :)

Sounds great though Eric.

[Simon Lloyd]

Hi Guy's, this mod is great but has me a little worried, i had an email today from a user who could not gain access, i checked and his ip isn't noted at honeypot, could someone help/explain so i am SURE that no real users are getting caught in future? details below:
HEADERGET / HTTP/1.1
Accept: image/gif, image/jpeg, image/pjpeg, image/pjpeg, application/xaml+xml, application/vnd.ms-xpsdocument, application/x-ms-xbap, application/x-ms-application, application/x-shockwave-flash, application/vnd.ms-excel, application/msword, application/vnd.ms-powerpoint, */*
Accept-Encoding: gzip, deflate
Accept-Language: en-us
Connection: keep-alive
Cookie: tccsessionhash=3fa5a7621ebcf4e360470468df3ff627; vbet_sessionUsed=1; tcclastvisit=1304375553; tcclastactivity=0; PHPSESSID=6beff1fb7aa112e8fa284b69284d36c6; BVGDU=http%3A//www.hyperpromote.com/tags/showdsnrsec1.html%3Fbvlocationcode%3D666098; BVGDT=21600; pBVPU=yes
Host: www.thecodecage.com
Referer: http://www.excel-it.com/index.html
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://bsalsa.com) ; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.5.30428; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; MS-RTC LM 8; .NET4.0C; .NET4.0E; InfoPath.3; msn OptimizedIE8;ENUS)
Via: 1.1 nap4-wsa2.boyd.net:80 (IronPort-WSA/7.1.0-307)
X-Imforwards: 20
URI is just /

PROTOCOL HTTP/1.1

METHOD GET

USERAGENTMozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://bsalsa.com) ; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.5.30428; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; MS-RTC LM 8; .NET4.0C; .NET4.0E; InfoPath.3; msn OptimizedIE8;ENUS)

KEY 17f4e8c8

IP 64.79.129.xxx (IP has been checked and not logged as bad)

Also Eric, under the IP there was no mention of UserId, should it have shown a userid as he is a valid member?

[error10]

Hi Guy's, this mod is great but has me a little worried, i had an email today from a user who could not gain access, i checked and his ip isn't noted at honeypot, could someone help/explain so i am SURE that no real users are getting caught in future? details below:
HEADERURI is just /

PROTOCOL HTTP/1.1

METHOD GET

USERAGENT

KEY 17f4e8c8

IP 64.79.129.xxx (IP has been checked and not logged as bad)

Also Eric, under the IP there was no mention of UserId, should it have shown a userid as he is a valid member?

This person had some junk installed on their computer that put the bsalsa.com string in their user agent. The user agent string remains even after they remove the malware. If they click the fix it yourself link, it tells them how to remove it.

[error10]

Currently I see a lot of users blocked by the accept issue and must disable BB because of this. The above function would allow me to keep BB running while the issue is addressed.

IMO this ticket about the accept issue should be reopened, as you are trying to resolve the issue:
http://trac.assembla.com/vb-bad-behavior/ticket/4

The one you posted in ticket 4 doesn't look like a legitimate user. Are you absolutely 100% certain that it is?

I plan on adding stopForumSpam actually. ;)

Please don't. I do not believe Stop Forum Spam is appropriate for use in Bad Behavior (http://bad-behavior.ioerror.us/2010/02/20/stop-forum-spam/), at least not until the service is changed to address my concerns.

[viper357]

I plan on adding stopForumSpam actually.

Please don't. I do not believe Stop Forum Spam is appropriate for use in Bad Behavior (http://bad-behavior.ioerror.us/2010/02/20/stop-forum-spam/), at least not until the service is changed to address my concerns.
I agree, I would much prefer the two mods to be kept separate.

[Simon Lloyd]

This person had some junk installed on their computer that put the bsalsa.com string in their user agent. The user agent string remains even after they remove the malware. If they click the fix it yourself link, it tells them how to remove it.Thanks for the prompt reply but where would they find the "fix it yourself" link?

[error10]

Thanks for the prompt reply but where would they find the "fix it yourself" link?

Hm, something obviously got lost in translation somewhere.

Anytime some request is blocked by Bad Behavior, a custom error page is displayed which contains some basic information, a technical support key, and a "fix it yourself" link the person can click on to get detailed information about their specific issue.

For instance, if someone's blocked by Project Honey Pot they go to the Project Honey Pot page for their IP address, where they can unblock themselves.

In your specific case, they go to a page where they can download a custom registry cleaning script for this specific issue (http://bad-behavior.ioerror.us/clean-post-platform.reg).

Anybody reporting that they were blocked by Bad Behavior should be able to provide the technical support key that was displayed. If there was no key, then they either weren't blocked by Bad Behavior, or didn't bother to read what was right in front of them.

[Eric]

Hi Guy's, this mod is great but has me a little worried, i had an email today from a user who could not gain access, i checked and his ip isn't noted at honeypot, could someone help/explain so i am SURE that no real users are getting caught in future? details below:
HEADERURI is just /

PROTOCOL HTTP/1.1

METHOD GET

USERAGENT

KEY 17f4e8c8

IP 64.79.129.xxx (IP has been checked and not logged as bad)

Also Eric, under the IP there was no mention of UserId, should it have shown a userid as he is a valid member?
The Cookie did not have a userid present, so that's why it didn't show.

The one you posted in ticket 4 doesn't look like a legitimate user. Are you absolutely 100% certain that it is?



Please don't. I do not believe Stop Forum Spam is appropriate for use in Bad Behavior (http://bad-behavior.ioerror.us/2010/02/20/stop-forum-spam/), at least not until the service is changed to address my concerns.

I agree, I would much prefer the two mods to be kept separate.
I've had a few requests for it, I may add it then have it disabled by default. I'll think about it :)

[error10]

I've had a few requests for it, I may add it then have it disabled by default. I'll think about it :)

If Stop Forum Spam gets their act together, then I'll add it in myself. But it's been over a year since I looked at it, and they've gone from having a manual removal form to no removal form at all. They claim it's broken; I suspect it was inundated with spam. This is the wrong direction, I think. Remember, with these blacklists I want to give legitimate people who wind up with a dirty IP address an easy way out.

I've tested a lot of blacklists over the years, and found that all of them block legitimate users from time to time simply because of aggressive IP reuse by ISPs. Since most of them are designed to stop email spam, it's OK if they have a more involved removal process, since it's generally only something the ISP will do, but for the purpose of securing a web site, removals have to be fast and easy. This is where Stop Forum Spam fails.

[Simon Lloyd]

Error10, thanks very much for the detailed explanation, as i've never been blocked by badbehaviour i would not have seen it, when i click the link in the logs for the key i have never seen a link or technical key :)

[error10]

Currently I see a lot of users blocked by the accept issue and must disable BB because of this. The above function would allow me to keep BB running while the issue is addressed.

I haven't forgotten about you. My simple idea for taking care of this issue wasn't so simple after all. This one check blocks a significant portion of harvesters, attack tools, etc., and my first pass at this - while it would let in all of these users - would also let in much of the traffic which was spiking your server load through the roof.

If you want to take that risk, I can send you a custom patch you can upload which will disable the Accept: test.

I also think that an option should exist to allow for registered users to bypass some or all of Bad Behavior's tests. A formal API for this is on the 3.0 roadmap, though I think Eric could whip up some hackery to add this in.

[error10]

Error10, thanks very much for the detailed explanation, as i've never been blocked by badbehaviour i would not have seen it, when i click the link in the logs for the key i have never seen a link or technical key :)

I should hope most people will never see it. :) In any case you can advise your user to click the fix it yourself link and to use the registry cleaner provided. (He will have to reboot the computer after using it, but the page also explains that.)

[Simon Lloyd]

Great, thanks for that, soooooooo one last question (i guess for Eric) where are these custom pages stored so we can make the message more prominant, add our forums css and maybe add a link to a helpdesk (this is how i was contacted by my user)?

[Simon Lloyd]

..I also think that an option should exist to allow for registered users to bypass some or all of Bad Behavior's tests. A formal API for this is on the 3.0 roadmap, though I think Eric could whip up some hackery to add this in.Sounds great but wouldn't that mean storing all useragents & IP's used by each user in the database?, i say this because BB would need to know to allow that user to even view the site in the first place, naturally if the user gets to login they've gone as far as the need to anyway. I'm not sure how you would police it without storing that added information for every user, unless i've misread how BB works.

[error10]

Hey, let's see if I can learn to multi-quote!

Great, thanks for that, soooooooo one last question (i guess for Eric) where are these custom pages stored so we can make the message more prominant, add our forums css and maybe add a link to a helpdesk (this is how i was contacted by my user)?

Right now Bad Behavior doesn't have any way to theme the page shown to blocked requests; it's all hard coded. Since it has to run under so many platforms, I basically just internalized everything, including the technical support pages. I'll make sure this gets on the roadmap.

Sounds great but wouldn't that mean storing all useragents & IP's used by each user in the database?, i say this because BB would need to know to allow that user to even view the site in the first place, naturally if the user gets to login they've gone as far as the need to anyway. I'm not sure how you would police it without storing that added information for every user, unless i've misread how BB works.

Ha, you've clearly seen to the root of the problem. So I guess that won't work very well, or at all. As you can see, sorting malicious actors from real people in real time is a rather hard problem.

[Simon Lloyd]

Again, thanks for the responses, i've tinkered ever so slightly with responses.inc.php to add the url to my helpdesk just before ', 'log' =>I assume the url won't be parsed in this fashion, it's not a problem if it doesn't.

I see that what you're trying to do is commendable and fantastic, you've saved my bandwidth usage no end (when next months revenue comes in i'll make a donation....already did to project honey pot as i thought that was yours, was still a worthy donation), what i do see is that if you did fill the request to have every visitors user agent...etc checked with projecthoneypot, htppBL and every entry in your own database to summise it's a registered user then the server load and resource usage would kill your forum (on a busy one anyway or if you are running VPS or VM's).

I believe the way you are tackling it is the most sensible, the honing of this software and minimising the effect on honest organics is definitely the way to go. I guess the only other option for "suspect" real users is to filter those through to another list or moderated usergroup so they can then be contacted through the forum for an organic response this way you would also capture any secondary ip they may be using to allow you to whitelist them, perhaps to this end is it possible to add to your roadmap 3.0 to have the whitelist (and maybe a blacklist) integrated into vBBB so that it can be edited directly in admincp>vb Bad Behaviour Options (probably one for Eric?)?

Anyway above all, another great big thanks to you guys, nominated MOTM

[error10]

Thanks for your compliments. :)

One thing to note: Bad Behavior is not intended to be a complete anti-spam solution; it should not be the only thing you run (http://bad-behavior.ioerror.us/documentation/spam-prevention-strategy/). Bad Behavior should also not do certain things and indeed, a close inspection of the code will reveal quite a few things which have been either partially implemented, or tested and found to not work and therefore disabled.

Bad Behavior is meant to block a majority of obvious spam, in order to reduce server load and reduce the amount of spam messages and registrations to a level that is manageable with more traditional tools. Because this is done by completely blocking the request and stopping vBulletin (or other software) from completing loading, it's simply not possible for me to do everything. Some things must be let through because I can't reliably distinguish them in real time.

Bad Behavior is also not meant to be a general purpose blacklisting tool, as a few people here have tried to use it. While it does contain an internal blacklist, these items are limited to well known malicious user-agents which scrape, harvest addresses, deliver spam or execute attacks. Things like ht:track and wget are intentionally not on the blacklist because many people want such software to visit their sites, and they are not designed as malicious tools. It's better to add such things to your local .htaccess (or equivalent) if you intend to block them. Perishable Press (http://perishablepress.com/) has some really good starting points.

[Alfa1]

The one you posted in ticket 4 doesn't look like a legitimate user. Are you absolutely 100% certain that it is?
I cleared my logs, but it was a registered member. I think it was a valid member.
Just to make sure, here are some more. All from valid members:

Please check out: OpenSearch (https://vborg.vbsupport.ru/showthread.php?t=119144&highlight=ncode)
By far most issues seem to be related to this.
Key: HTTP Response: 403
Explanation: An invalid request was received from your browser. This may be caused by a malfunctioning proxy server or browser privacy software.
Log Message: Required header 'Accept' missing
UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.60 Safari/534.24
URI: /forum/misc.php?do=page&template=ncode_opensearch
Entity:
Headers: GET /forum/misc.php?do=page&template=ncode_opensearch HTTP/1.1
Host: www.my-forum.com
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.60 Safari/534.24
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: [omitted data] bbcybfr_redtopage=newthread.php%3Fdo%3Dnewthread%2 6f%3D123; [omited data]

Key: HTTP Response: 403
Explanation: An invalid request was received from your browser. This may be caused by a malfunctioning proxy server or browser privacy software.
Log Message: Required header 'Accept' missing
UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_7) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.57 Safari/534.24
URI: /forum/misc.php?do=page&template=ncode_opensearch
Entity:
Headers: GET /forum/misc.php?do=page&template=ncode_opensearch HTTP/1.1
Host: www.my-forum.com
Connection: keep-alive
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_7) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.57 Safari/534.24
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-GB,en-US;q=0.8,en;q=0.6
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: [data omitted]


Key: HTTP Response: 403
Explanation: An invalid request was received from your browser. This may be caused by a malfunctioning proxy server or browser privacy software.
Log Message: Required header 'Accept' missing
UserAgent: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.60 Safari/534.24
URI: /forum/misc.php?do=page&template=ncode_opensearch
Entity:
Headers: GET /forum/misc.php?do=page&template=ncode_opensearch HTTP/1.1
Host: www.my-forum.com
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.60 Safari/534.24
Accept-Encoding: gzip,deflate,sdch
Accept-Language: it-IT,it;q=0.8,en-US;q=0.6,en;q=0.4
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: [data omitted]

Key: HTTP Response: 403
Explanation: An invalid request was received from your browser. This may be caused by a malfunctioning proxy server or browser privacy software.
Log Message: Required header 'Accept' missing
UserAgent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.60 Safari/534.24
URI: /forum/local_links.php?action=jump&catid=57&id=7208
Entity:
Headers: GET /forum/local_links.php?action=jump&catid=57&id=7208 HTTP/1.1
Host: www.my-forum.com
Connection: keep-alive
Referer: http://www.my-forum.com/forum/local_links.php?catid=57
User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.60 Safari/534.24
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: [data omitted]

[Alfa1]

The best way to handle bypassing members would be to only run the plugin for guests - which I can code that in.
That would be a good option to have.

Right now Bad Behavior doesn't have any way to theme the page shown to blocked requests; it's all hard coded. Since it has to run under so many platforms, I basically just internalized everything, including the technical support pages. I'll make sure this gets on the roadmap.
It would be very useful if legitimate members who face difficulties can be pointed to our helpdesk url.

[Simon Lloyd]

Again, thanks for the responses, i've tinkered ever so slightly with responses.inc.php to add the url to my helpdesk just before I assume the url won't be parsed in this fashion, it's not a problem if it doesn't.


It would be very useful if legitimate members who face difficulties can be pointed to our helpdesk url.See my quote above? ;)

HTTP Response: 403
Explanation: You do not have permission to access this server. Before trying again, run anti-virus and anti-spyware software and remove any viruses, malware or spyware from your computer.If you need further help then contact Our HelpDesk (http://mysite.com)
Log Message: IP address found on http:BL blacklist

[Alfa1]

Thanks! I assume I can just use html in there?

[Simon Lloyd]

I did for the link and it parsed ok :)

[Simon Lloyd]

Any ideas why this one has been blocked:Header
POST /forumz/login.php?do=login HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-us
Cache-Control: no-cache
Connection: Keep-Alive
Cookie: bb2_screener_=1304316965+216.67.121.73; bb2_screener_=1304476965+74.124.89.49; tccsessionhash=ac18ec200b202084e3f43421bfd41ebd; vbet_sessionUsed=1; __utmc=118899148; __utma=118899148.893063458.1303547625.1304477337.1 304569993.33; __utmz=118899148.1304317329.30.5.utmcsr=thecodecag e.com|utmccn=(referral)|utmcmd=referral|utmcct=/; __utmv=118899148.usergroup-1-Unregistered%20%2F%20Not%20Logged%20In; tcclastvisit=1303548761; tcclastactivity=0; __utmb=118899148.2.10.1304569993; tccsessionhash=a0bc27efc24ca21550367fd6d71b19c8; tccthread_lastview=f7b50b30415971c4d96c712d31d24b2 7503a8bc1a-1-%7Bi-208810_i-1304249678_%7D; HESK=19ea9b13f1cc8ca4ffd7e262f37d54f0; tcclastvisit=1303548761; tcclastactivity=0; tccuserlgv=2
Host: www.thecodecage.com
Referer: http://thecodecage.com/
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.6; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.3; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
KEY
b40c8ddc
User Agent
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.6; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.3; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
METHOD
POST
URI
forumz/login.php?do=login
ENTITY
vb_login_username: XxXxX
vb_login_password:
s:
securitytoken: guest
do: login
vb_login_md5password: 7327xxxxxxxxxxxxxxxxxxxxxxxxxxxb90
vb_login_md5password_utf: 7327xxxxxxxxxxxxxxxxxxxxxxxxxb90
ajax: 0
It's a genuine user who has tried from 2 different IP's but still gets blocked.

[Eric]

Any ideas why this one has been blocked:It's a genuine user who has tried from 2 different IP's but still gets blocked.
That appears to be the POST/GET error. Can you verify that the mod is injecting the javascript from BB into your header? If I'm not mistaken, that not being present can help lead to this error.

[Eric]

Alfa, could you go into more detail on this please? http://trac.assembla.com/vb-bad-behavior/ticket/8

EDIT: and http://trac.assembla.com/vb-bad-behavior/ticket/12#comment:1

[Alfa1]

I do not seem to have commenting or editing functionality on trac. So here goes:

Send registered member explanation how to resolve blacklisting (http://trac.assembla.com/vb-bad-behavior/ticket/12)
Yes, an email would be more effective than a PM.

Trace IP directly from the log. (http://trac.assembla.com/vb-bad-behavior/ticket/8)
In the log hotlink the IP of the user. The link should point to a whois for the user. For example /admincp/usertools.php?do=gethost&ip=xx.xx.xx.xx
Or a better whois service like http://who.is/whois-ip/ip-address/xx.xx.xx.xx/

Alert the admin which members have been blocked by BB and why. (http://trac.assembla.com/vb-bad-behavior/ticket/7)
For sites with a lot of traffic, sending a notification such as this by PM may be overkill - then again, not sure what other option would be available.
Such notification should be sent max once per X days and should list all blocked members since the last notification.
It would be useful to include some additional information like join date, post count and usergroup of the member. This makes it easier to see if the user is a legitimate user.

[Alfa1]

I am having no success with adding these bots to my blacklist:
Alexa
Artabus
BoardPulse
Deepnet Explorer
Radian6 FeedFetcher
Wget

Cna someone tell me what exactly I need to add to my blacklist.inc?

I see that adding 'Alexa' to my blacklist also blocks users with the alexa toolbar. It seems to me that if I want to allow users with the alexa toolbar installed, I will not be able to block the alexa crawler?

[tpearl5]

In less than a day my log is already over 5,000 entries! A lot of them are "Required header 'Accept' missing". I've had one member report seeing strange errors, but I can't pinpoint the user in the logs.

I think an IP search for the logs may be a useful addition.

[Alfa1]

I am having no success with adding these bots to my blacklist:
Alexa
Artabus
BoardPulse
Deepnet Explorer
Radian6 FeedFetcher
Wget

Cna someone tell me what exactly I need to add to my blacklist.inc?

I see that adding 'Alexa' to my blacklist also blocks users with the alexa toolbar. It seems to me that if I want to allow users with the alexa toolbar installed, I will not be able to block the alexa crawler?
I see that alexa, deepnet explorer, radian6 feedfetcher and wget do get blocked, but are also listed on my spiders who visited list. Should that be?

[Simon Lloyd]

That appears to be the POST/GET error. Can you verify that the mod is injecting the javascript from BB into your header? If I'm not mistaken, that not being present can help lead to this error.Eric i'd love to confirm this .....could you tell me how?

[Simon Lloyd]

ahh, just going to www.thecodecage.com/forumz and right click view source shows<script type="text/javascript">
<!--
function bb2_addLoadEvent(func) {
var oldonload = window.onload;
if (typeof window.onload != 'function') {
window.onload = func;
} else {
window.onload = function() {
oldonload();
func();
}
}
}

bb2_addLoadEvent(function() {
for ( i=0; i < document.forms.length; i++ ) {
if (document.forms[i].method == 'post') {
var myElement = document.createElement('input');
myElement.setAttribute('type', 'hidden');
myElement.name = 'bb2_screener_';
myElement.value = '1304785371 2.127.13.238';
document.forms[i].appendChild(myElement);
}
}
});
// --></script>
is this what you meant?

[Lee G]

Just checked my logs and another google bot got caught

User agent
Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13

ip
66.249.82.129

Full ip range
66.249.64.0/19

And another google ip range for google to whitelist
64.233.160.0/19

Bot from ip 64.233.172.18 got caught

[Simon Lloyd]

Hi, i've whitelisted the users IP(s) but they are still being blocked "Post more than two days after Get" here's the headerPOST /forumz/login.php?do=login HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-us
Cache-Control: no-cache
Connection: Keep-Alive
Cookie: bb2_screener_=1304316965+216.67.121.73; bb2_screener_=1304647462+74.124.87.161; tccsessionhash=62a00b28d319ba412c2ffd704b22a856; vbet_sessionUsed=1; __utmc=118899148; __utma=118899148.893063458.1303547625.1304569993.1 304647832.34; __utmz=118899148.1304317329.30.5.utmcsr=thecodecag e.com|utmccn=(referral)|utmcmd=referral|utmcct=/; __utmv=118899148.usergroup-1-Unregistered%20%2F%20Not%20Logged%20In; tcclastvisit=1303548761; tcclastactivity=0; tccsessionhash=eea70a6bbfea2bdd00a4029b51c8e209; tccthread_lastview=f7b50b30415971c4d96c712d31d24b2 7503a8bc1a-1-%7Bi-208810_i-1304249678_%7D; HESK=19ea9b13f1cc8ca4ffd7e262f37d54f0; tcclastvisit=1303548761; tcclastactivity=0; tccuserlgv=2
Host: www.thecodecage.com
Referer: http://thecodecage.com/
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.6; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.3; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
I do notice two differences in IP (one reversed), am i right? i dont understand anything of the above so im guessing but i thought there may be a problem here Cookie: bb2_screener_=1304316965+216.67.121.73; bb2_screener_=1304647462+74.124.87.161;

[Eric]

Hi, i've whitelisted the users IP(s) but they are still being blocked "Post more than two days after Get" here's the headerI do notice two differences in IP (one reversed), am i right? i dont understand anything of the above so im guessing but i thought there may be a problem here

Two different IP's - have the user delete their cookies and see if it still happens.

[Eric]

I see that alexa, deepnet explorer, radian6 feedfetcher and wget do get blocked, but are also listed on my spiders who visited list. Should that be?
It's possible they are picked up as spiders visited before BB is ran.

[Eric]

Just checked my logs and another google bot got caught

User agent
Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13

ip
66.249.82.129

Full ip range
66.249.64.0/19

And another google ip range for google to whitelist
64.233.160.0/19

Bot from ip 64.233.172.18 got caught
Hmm. Will add those to the default list.

[Eric]

The user anonymous has been granted the permission TICKET_MODIFY.

You should be able to now.

And thanks for going into a little bit more detail on those for me :)

I do not seem to have commenting or editing functionality on trac. So here goes:

Send registered member explanation how to resolve blacklisting (http://trac.assembla.com/vb-bad-behavior/ticket/12)
Yes, an email would be more effective than a PM.

Trace IP directly from the log. (http://trac.assembla.com/vb-bad-behavior/ticket/8)
In the log hotlink the IP of the user. The link should point to a whois for the user. For example /admincp/usertools.php?do=gethost&ip=xx.xx.xx.xx
Or a better whois service like http://who.is/whois-ip/ip-address/xx.xx.xx.xx/

Alert the admin which members have been blocked by BB and why. (http://trac.assembla.com/vb-bad-behavior/ticket/7)

Such notification should be sent max once per X days and should list all blocked members since the last notification.
It would be useful to include some additional information like join date, post count and usergroup of the member. This makes it easier to see if the user is a legitimate user.

[Simon Lloyd]

Eric, as always, really appreciate the reply, this user is one of my more valuable ones so need to get this resolved. I have asked them to clear their cookies next time they log in, however this is happening to them from 3 different machines, this person is one of two actual members to be caught up although there are many "users awaiting confirmation" getting caught although i can't decipher how many of these are genuine or not without going through the many thousands of entries one by one and checking their IP's against Project Honeypot.

I do appreciate everything you are trying to do to resolve this.

[Alfa1]

It's possible they are picked up as spiders visited before BB is ran.
I use Paul M's Track Guest Visits (https://vborg.vbsupport.ru/showthread.php?t=201214) to monitor spiders.

You should be able to now.

And thanks for going into a little bit more detail on those for me :)
Could you give the user Alfa1 permission? I have an account at trac.

Some brainstorming:
An important factor in regards to bots being malicious or not, is if they respect robots.txt or not. If I want to block a bot, the first thing I do is disallow it in robots.txt
If after a week I still find it on my site (and therefore the bot has not respected robots.txt), then I blacklist the bot in BB.

I wonder if it would be a good idea to automatically blacklist bots that disrespect robots.txt ?


Does anyone (especially error10) know if the spiderlist.xml provided by Mosh is complete enough for the purposes of keeping an eye on malicious bots?
See: http://www.wolfshead-solutions.com/display-spiders

[Simon Lloyd]

Hi Eric, an update on the blocked users, it seems that they are getting as far as the login, they get the vb welcome message on logon, nothing at all from BB but after the welcome screen they are shown as though they have not logged on i.e username and password boxes remain blank and they remain in the "Unregistered/Not logged on" usergroup, however BB shows the results i have previously posted?

[Eric]

Hi Eric, an update on the blocked users, it seems that they are getting as far as the login, they get the vb welcome message on logon, nothing at all from BB but after the welcome screen they are shown as though they have not logged on i.e username and password boxes remain blank and they remain in the "Unregistered/Not logged on" usergroup, however BB shows the results i have previously posted?
Hmm. I'll be releasing an update soon that will allow you to skip members in the BB processing. If you want to try it out (** UNTESTED AS OF YET **) http://trac.assembla.com/vb-bad-behavior/changeset/50/trunk/vb3/product-vb_badbehavior.xml?old=44&old_path=trunk%2Fvb3%2Fproduct-vb_badbehavior.xml

[Simon Lloyd]

Thanks Eric, installed, i have cleared my logs and will download them as a csv after around 5 days, you're quite welcome to have it for your analysis.

[Simon Lloyd]

Eric, the user has reported back:
Tried from home computer. Different result. Got the following:
Unable to add cookies, header already sent.
File: /home/thecodec/public_html/forumz/global.php(1091) : eval()'d code
Line: 95

[Simon Lloyd]

In fact just noticed that that is showing all the time at the top of the forum so i have reverted back to bb1.0.4

[tpearl5]

While one of my moderators was removing a thread she got this vbulletin message on login.php?do=login:

"Your submission could not be processed because a security token was missing"

[Simon Lloyd]

Hi Eric, i still cannot get your new beta version working so am running the previous update, hwoever we have finally got to the bottom of that valued user not being able to log on with the help firstly of Trend Micro HouseCall free online scan found a lot of malware on her PC, it had also crippled her useragent string which we discovered by getting to go here http://user-agents.my-addr.com/user_agent_request/user_agent_examples-and-user_agent_types.php and see what it said her useragent was whilst there, now for some reason in the forum here useragent showed as Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.6; .NET CLR 1.1.4322; .NET CLR
but on visiting that site its showed Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.6; .NET CLR 1.1.4322; .NET CLR 2.0.50727;http://www.hyperpromote.com/tags/showdsnrsec1.html%3Fbvlocationcode%3D666098; BVGDT=21600; pBVPU=yes; InfoPath.3; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
so there was much more to the string and it had been corrupted, even though i was whitelisting all her IPs it was still denying her. Now we know what the issue is we need to find out how to fix her UA string.

Thanks for your help :)

[Eric]

Eric, the user has reported back:

In fact just noticed that that is showing all the time at the top of the forum so i have reverted back to bb1.0.4
Argh, looks like a rogue ')' in the plugin code, will fix. :)

While one of my moderators was removing a thread she got this vbulletin message on login.php?do=login:

"Your submission could not be processed because a security token was missing"
This mod doesn't mess with the security token, so I am not sure why they would get that error.

Hi Eric, i still cannot get your new beta version working so am running the previous update, hwoever we have finally got to the bottom of that valued user not being able to log on with the help firstly of Trend Micro HouseCall free online scan found a lot of malware on her PC, it had also crippled her useragent string which we discovered by getting to go here http://user-agents.my-addr.com/user_agent_request/user_agent_examples-and-user_agent_types.php and see what it said her useragent was whilst there, now for some reason in the forum here useragent showed as but on visiting that site its showed so there was much more to the string and it had been corrupted, even though i was whitelisting all her IPs it was still denying her. Now we know what the issue is we need to find out how to fix her UA string.

Thanks for your help :)
Ahh, that explains it. For MSIE, I think you can fix the user agent by editing the registry. I'll look into that for you.

[Eric]

I use Paul M's Track Guest Visits (https://vborg.vbsupport.ru/showthread.php?t=201214) to monitor spiders.


Could you give the user Alfa1 permission? I have an account at trac.

Some brainstorming:
An important factor in regards to bots being malicious or not, is if they respect robots.txt or not. If I want to block a bot, the first thing I do is disallow it in robots.txt
If after a week I still find it on my site (and therefore the bot has not respected robots.txt), then I blacklist the bot in BB.

I wonder if it would be a good idea to automatically blacklist bots that disrespect robots.txt ?


Does anyone (especially error10) know if the spiderlist.xml provided by Mosh is complete enough for the purposes of keeping an eye on malicious bots?
See: http://www.wolfshead-solutions.com/display-spiders
Did you register? I don't think assembla allows registration, didn't see any accounts. Anonymous and registered users have the permission now either way though.

As far as blocking bots who disrespect robots.txt - I have an idea for that.

[Eric]

Just checked my logs and another google bot got caught

User agent
Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13

ip
66.249.82.129

Full ip range
66.249.64.0/19

And another google ip range for google to whitelist
64.233.160.0/19

Bot from ip 64.233.172.18 got caught
Bad Behavior Core will need to be updated to work around this it looks like. It checks for "Googlebot", not just "Google". Will email error10

[Alfa1]

Did you register? I don't think assembla allows registration, didn't see any accounts. Anonymous and registered users have the permission now either way though.

As far as blocking bots who disrespect robots.txt - I have an idea for that.
Yes, I registered. I am afraid that I have permission to reopen and edit tickets. See: http://trac.assembla.com/vb-bad-behavior/ticket/4

Im glad that to see that 1.0.5 has Setting to make registered users bypass BB (http://trac.assembla.com/vb-bad-behavior/ticket/13) implemented.

[Lee G]

Hi Eric
Is there any news on the updates for this
Id the stop forum spam integration still going ahead or has that idea been shelved

[Alfa1]

I look forward to the next version as well. It will resolve the issue of 80 legitimate members being blocked every day.

[raj4x]

Hi Eric,

Looks awesome.

Forgive my ignorance but how does vB Bad Behavior differ from Spam-O-Matic?

Cheers, Raj.

[Lee G]

From memory spam o matic works by connecting to different databases and checking ip when people hit the register button

This works even further up the line

It checks user agents, connection methods and the person gets an error page
Along with checking the ips against project honey pot
Using this, they spammers / scrapers / email harvesters dont even get a look at any pages, let alone being able to hit the register button

Its worth running the two side by side. I run the original stop forum spam from on here, along side this. I get very few spam registrations these days

I also found on my forums that htaccess user agent bans were not working
But by putting the same user agents into the blacklist file on the bad behavior mod, I can get around the problem. By adding all the search engines that are genuine to the whitelist file, you dont inadvertently block the likes of Google etc

One of the first things you notice is the lack of spam hitting your email each day

[Eric]

Still working on 1.0.5. I never heard back from error10, so I modified a core bad-behavior file myself for the Google Web Preview issue. I'm estimating a release sometime in June.

Currently:

Version 1.0.5, 06/??/2011
- Added option for bypassing users/members.
o If the visitor is a user, and is in usergroup 5, 6, or 7 (admin/mod/super mod) - Bad Behavior is bypassed.
- Modified bad-behavior core to check for Google Web Preview
o file edited: /includes/bad-behavior/core.inc.php
- Added a link beside the IP address in the log for WhoIs.


Code Changes: http://trac.assembla.com/vb-bad-behavior/changeset?new=60%40trunk%2Fvb3&old=44%40trunk%2Fvb3

[Alfa1]

Could you please push out a small release inbetween, so that my members are no longer blocked by BB?

[BirdOPrey5]

Still working on 1.0.5. I never heard back from error10, so I modified a core bad-behavior file myself for the Google Web Preview issue. I'm estimating a release sometime in June.

Currently:

Version 1.0.5, 06/??/2011
- Added option for bypassing users/members.
o If the visitor is a user, and is in usergroup 5, 6, or 7 (admin/mod/super mod) - Bad Behavior is bypassed.
- Modified bad-behavior core to check for Google Web Preview
o file edited: /includes/bad-behavior/core.inc.php
- Added a link beside the IP address in the log for WhoIs.


Code Changes: http://trac.assembla.com/vb-bad-behavior/changeset?new=60%40trunk%2Fvb3&old=44%40trunk%2Fvb3

Not installed yet but following closely. For the usergroup bypass- will this be an option in the settings or will 5, 6, and 7 be hard coded? I hope I can enter additional usergroups to bypass in the settings- both primary and secondary.

Thanks.

[Eric]

Could you please push out a small release inbetween, so that my members are no longer blocked by BB?
I will try to get 1.0.5 out as soon as possible. I am rather busy right now, just started my first two college courses on Monday :)

Not installed yet but following closely. For the usergroup bypass- will this be an option in the settings or will 5, 6, and 7 be hard coded? I hope I can enter additional usergroups to bypass in the settings- both primary and secondary.

Thanks.
5, 6, 7 will be hard coded but you will also be able to enter additional usergroups :)

[Lee G]

Thanks for the update Eric
Good luck with the studies

[Eric]

Version 1.0.5, 05/26/2011

Added option for bypassing users/members.
If the visitor is a user, and is in usergroup 5, 6, or 7 (admin/mod/super mod) - Bad Behavior is bypassed.
Modified bad-behavior core to check for Google Web Preview

file edited: /includes/bad-behavior/core.inc.php

Added a link beside the IP address in the log for WhoIs.
Changes: http://trac.assembla.com/vb-bad-behavior/changeset?new=62%40trunk%2Fvb3&old=44%40trunk%2Fvb3

[Lee G]

Cheers for that Eric, top man

Made similar changes to those posted by BadgerDog on the VB4 version, for my registered member groups
https://vborg.vbsupport.ru/showpost.php?p=2200528&postcount=48

[Simon Lloyd]

Eric, installed and appears to be working fine :)

[Alfa1]

Thanks Eric!

[viper357]

Alfa1, please let us know if you are still getting members blocked, I've been holding off installing this until that issue has been resolved, thanks. :)

[Lee G]

You can edit the xml to include the user groups you want to bypass bb
Look for the line
else if (is_member_of($vbulletin->userinfo, 5, 6, 7
And add the extra user groups to that

Then upload the xml in the normal way

[Eric]

I just wanted to note... that you do not necessarily have to edit the usergroups in the file. You can limit it to members with posts less than the post count option. :)

[Alfa1]

Alfa1, please let us know if you are still getting members blocked, I've been holding off installing this until that issue has been resolved, thanks. :)
It works like a charm! Go for it.
I have made members immune that have registered more than 30 days ago.

I just wanted to note... that you do not necessarily have to edit the usergroups in the file. You can limit it to members with posts less than the post count option. :)
The admincp option is for days registered. Is there a post count option as well?

[Eric]

ugh, sorry. My bad. I meant days registered, don't know what I was thinking. Would ya'll like a post count option too?

[Alfa1]

For me the days registered option is fine.

[Simon Lloyd]

Eric, from my point of view a post count check would be great as, like all forums running this, there will have been many that signed up a long time ago that are bots....etc, so i'd like to say "Is Member AND Has PostCount >...."

[viper357]

Does this mod work ok with Tapatalk?

[Lee G]

Are the logs set to clear every 7 or 30 days?
Im showing 87,310 total log entries at present
But I do have a lot more user agents being banned than the standard list
Thats a lot of morons with wasted bandwidth trying to hit and scrape

[Eric]

Are the logs set to clear every 7 or 30 days?
Im showing 87,310 total log entries at present
But I do have a lot more user agents being banned than the standard list
Thats a lot of morons with wasted bandwidth trying to hit and scrape
Within BB core, it is hard-coded to prune every 7 days. I have never actually tested to see if it does that though, to be honest.

[Eric]

Does this mod work ok with Tapatalk?
I have no way to test Tapatalk unfortunately, so I do not know.

[Eric]

Eric, from my point of view a post count check would be great as, like all forums running this, there will have been many that signed up a long time ago that are bots....etc, so i'd like to say "Is Member AND Has PostCount >...."

For me the days registered option is fine.
I will leave the days registered option, but I may add the post count option as well and have a way to enable or disable it.

[Lee G]

Does this mod work ok with Tapatalk?

Try accessing my forum with tapatalk to see if it works
I dont use a mobile to access mine, but it is installed on there
Link to the forum is in my signature

[viper357]

Try accessing my forum with tapatalk to see if it works
I dont use a mobile to access mine, but it is installed on there
Link to the forum is in my signature
Nope, it says "This forum appears to have deactivated tapatalk".
I just enabled this mod and within 5 minutes I noticed in the logs what appears to be blackberry users logging on with Tapatalk were being blocked, so I've disabled it again as I have a large amount of users that access my forums via Tapatalk.

[Eric]

Nope, it says "This forum appears to have deactivated tapatalk".
I just enabled this mod and within 5 minutes I noticed in the logs what appears to be blackberry users logging on with Tapatalk were being blocked, so I've disabled it again as I have a large amount of users that access my forums via Tapatalk.
Would you mind providing me with user agent strings, etc for those users? If so, I may be able to create a whitelist for Tapatalk/mobile phones/etc

[viper357]

"user agent strings" - Please excuse me being an idiot here but I'm assuming this is the column in the logs that says User Agent? If so, they are all blank.

I can give you the info from the headers column, if that will help? I really don't know what most of this all means.:p

These first two are accessing the forum via Tapatalk, the "mobiquo" is the Tapatalk directory/mod.

POST /mobiquo/mobiquo.php HTTP/1.0
Accept: */*
Connection: keep-alive
Content-Length: 267
Host: www.marineaquariumsa.com
Cache-Control: max-age=259200
Via: BISB_3.5.1.76, 1.1 pmds152.bisb2.blackberry:3128 (squid/2.7.STABLE7)

POST /mobiquo/mobiquo.php HTTP/1.0
Accept: */*
Connection: keep-alive
Content-Length: 267
Host: www.marineaquariumsa.com
Cache-Control: max-age=259200
Via: BISB_3.5.1.76, 1.1 pmds252.bisb2.blackberry:3128 (squid/2.7.STABLE7)

This one looked like normal forum access but also says blackberry.

GET /showthread.php?p=470371#post470371/external.php?type=RSS2&forumids=219 HTTP/1.0
Accept: */*
Connection: keep-alive
Host: www.marineaquariumsa.com
Cache-Control: max-age=259200
Via: BISB_3.5.1.71, 1.1 pmds196.bisb4.blackberry:3128 (squid/2.7.STABLE7)

[BirdOPrey5]

I know tapatalk does something fishy with the user agent- it's caused issues with other security mods before.

[bulbasnore]

Tapatalk is big for us. That is definitely something to continue pursuing.

[Teemberland]

I was gonna install this, but I found out it doesn't work with Tapatalk. Update would be nice. :) Thank you.

[Alfa1]

These first two are accessing the forum via Tapatalk, the "mobiquo" is the Tapatalk directory/mod.
To resolve this talpatalk issue, you will likely just have to whitelist the url
/mobiquo/mobiquo.php

At the bottom of the file /includes/whitelist.ini add:
url[] = "/mobiquo/mobiquo.php"

That's it.

[carsafety]

Anyone know a similar whitelist for ForumRunner?

Also, has anyone who installed BadBehavior noticed a significant drop in Google Adsense, Amazon or revenue from other affiliates?

[Simon Lloyd]

I have installed it and it's had no impact on my revenue at all :)

[ClemsonJeeper]

Please add a whitelist for Forum Runner as well.

All requests come through

<base_forum_url>/forumrunner/request.php
<base_forum_url>/forumrunner/ad.php
<base_forum_url>/forumrunner/image.php
<base_forum_url>/forumrunner/detect.js

Thanks!

[Lee G]

Also, has anyone who installed BadBehavior noticed a significant drop in Google Adsense, Amazon or revenue from other affiliates?

What you should see is an increase in revenue
If you are lucky and it blocks a lot of bad or unwanted hits.
The less hits that people see your your adds on, should increase the views to clicks ratio and increase your earnings marginally

I block a lot more hits by adding extra user agents to one file with a reasonable amount of success

What would be a nice feature would be an option to block ips if they hit x amount of times with a banned user agent. 7 day, 14 day 30day options etc

The normal person will see the page of doom and either go away or look into solving the problem if they want to see the page.
The morons running these spam / scraping scripts will hit constantly
One which hit me this week, hit from at least 50 user agents. Most of which I have banned
I find it a regular thing to see 100 successive hits come in from a banned user agent
Then they come back again later with another user agent when they find that one failed.

[Alfa1]

Please add a whitelist for Forum Runner as well.
Is it needed to whitelist it? Is BB blocking anything in Forum Runner?

[viper357]

To resolve this talpatalk issue, you will likely just have to whitelist the url
/mobiquo/mobiquo.php

At the bottom of the file /includes/whitelist.ini add:
url[] = "/mobiquo/mobiquo.php"

That's it.
Thanks for that, but is there a way to whitelist all BlackBerry IP's? This mod is preventing anybody using a blackberry from accessing the forum. Thanks.

[Alfa1]

It does not block IPs, but user agents. But yeah, you can just whitelist anything blackberry if you want.

I'm not 100% sure about this one. Try:

In whitelist.ini

under
[useragent]
useragent[] = "Mozilla/4.0 (It's me, let me in)"
useragent[] = "Mozilla/4.0 (vBSEO; http://www.vbseo.com)"
useragent[] = "BoardTracker (http://www.boardtracker.com/spider.html) (Mozilla/4.0 compatible; MSIE 6.0; Linux CentOS;)"ADD:
useragent[] = "Blackberry (Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.9.0.8) (http://www.blackberry.net/go/mobile/profiles/uaprof/8530_evdo/5.0.0.rdf)"

Eric:
I have 286.000 entries in my logs. Its getting difficult to browse through this. Are entries really deleted from the log after 7 days or did BB block 286.000 attempts in the last week?

It would be very helpful to get an overview of registered members that were blocked. BB enables me to notify my members of troyans and other malware on their PC. So that after the removal of this malware they can freely browse my site. But there is no way that I can find time to review 286.000 entries per week.

Especially the entries that contain vb_login_username:

The IP whois does not work. It links to a white screen. Consider to use another whois service like http://whois.domaintools.com/

[Lee G]

I find the who is bit annoying
It seems to add more nloat and you tend to miss any members that have been inadvertently caught by mistake

Im also finding a lot of ips get through that are listed on Project Honeypot

[Alfa1]

I see that whitevector (http://www.whitevector.com) is hitting my website around 50.000 times a week, while its blacklisted in robots.txt
Im not sure about the exact number as I can't count it. Is there any way to count this?
This is a very serious amount and explains why my site was having such problems.

Does this count as a cyber attack? If it qualifies as a cyber attack, then I can press charges against them.

[Lee G]

Whitevector are not a robot as such
They are a corporate scraper, so in their tiny minds, do not need to obey robots text
The extortionate amounts they charge for scraping your site would dwindle if they did

Simple block in the first half of the blacklist file
"Whitevector Crawler", //

[Alfa1]

I had already blocked whitevector. (Right after installing BB) But whitevector makes up a substantial part of the BB logs. Its quite bizarre. This might explain why turning on BB had such a dramatic positive effect on my sites performance.

Well, I hope that their tiny minds can understand the letter that my lawyer is going to send to them. if I don't skip that part and file charges straight away.

[Lee G]

Another one to add to the second half is this

"SiteBot/0.1", //

Another scraper with a bad reputation being run out of the Ukraine and just about anywhere else the morons can hit with

2500 pages hit by that one yesterday in a short space of time

[carsafety]

Please add a whitelist for Forum Runner as well.


Thanks!

I've just enabled this mod to test it. No issues with ForumRunner so far, at least on the iPhone app. Apparently, Tapatalk does something different inherently that needs whitelisted.

Two questions for the devs or anyone who knows:

1) If I set "Bypass Users" to "Yes" and "Join Date" to 0, does that mean it will bypass all registered users?

2) For "Operating Mode", is normal mode "Yes" or is strict mode "Yes"?

I'll report on traffic and affiliate revenue in a week or so.

[carsafety]

Another one to add to the second half is this

"SiteBot/0.1", //

Another scraper with a bad reputation being run out of the Ukraine and just about anywhere else the morons can hit with

2500 pages hit by that one yesterday in a short space of time

Where do you enter additional blocks?

[Alfa1]

1) If I set "Bypass Users" to "Yes" and "Join Date" to 0, does that mean it will bypass all registered users?
That should be correct.

2) For "Operating Mode", is normal mode "Yes" or is strict mode "Yes"?
Normal mode should be set to yes, not strict mode.

[Alfa1]

Where do you enter additional blocks?
In includes/bad-behavior/blacklist.inc.php

[Lee G]

In the blacklist.inc.php you will see it split in two halfs
First section is starts with
Second half is anywhere in the user agent

The one I listed goes in the second half

With a bit of tweaking on that file you can drop out a lot of out the box xrummer user agents ;)

[carsafety]

In the blacklist.inc.php you will see it split in two halfs
First section is starts with
Second half is anywhere in the user agent

The one I listed goes in the second half

With a bit of tweaking on that file you can drop out a lot of out the box xrummer user agents ;)

Could you please post your file contents or perhaps send them to me via PM?

[Lee G]

This will be a good starting point
http://www.thespainforum.com/f379/bad-behavior-blacklist-306359/

PM me on there if you find you become instantly moderated
I moderate some countries straight off

If you use it, white list all the google ips etc. Just to be safe

My own black list changes on a daily basis

What I also do know is keep an eye on my logs
Project honey pot is good. But I catch a lot that have not been caught yet
Today I had one moron hit 42 times from different ips on a banned user agent that were not listed there
All they have to do is rotate the user agents and they are happily scraping your content

I also work on the idea of banning some agents that they use a lot and at the same time, might block a genuine hit or three in the process. I had a case where I was taking 200 bad hits a day on one agent and five good ones. It was a no brainer to ban it

Things like Opera auto update. If a person aint got the brains to update opera, they aint got the brains to use a forum

[viper357]

Will this mod block people that browse from work through company proxies?

[carsafety]

This will be a good starting point
http://www.thespainforum.com/f379/bad-behavior-blacklist-306359/

PM me on there if you find you become instantly moderated
I moderate some countries straight off

If you use it, white list all the google ips etc. Just to be safe

My own black list changes on a daily basis

What I also do know is keep an eye on my logs
Project honey pot is good. But I catch a lot that have not been caught yet
Today I had one moron hit 42 times from different ips on a banned user agent that were not listed there
All they have to do is rotate the user agents and they are happily scraping your content

I also work on the idea of banning some agents that they use a lot and at the same time, might block a genuine hit or three in the process. I had a case where I was taking 200 bad hits a day on one agent and five good ones. It was a no brainer to ban it

Things like Opera auto update. If a person aint got the brains to update opera, they aint got the brains to use a forum

Where do I find the logs? I don't see a log file anywhere obvious.

Also, aren't the Google IPs inherently whitelisted or do I have to go in and add them?

Sorry for the dumb questions, just trying to get this thing set up to a modest level of security without a lot of time I have to spend researching or editing.

[carsafety]

Eric, from my point of view a post count check would be great as, like all forums running this, there will have been many that signed up a long time ago that are bots....etc, so i'd like to say "Is Member AND Has PostCount >...."

+1

My new registrations are pretty well filtered, so I'd like to allow members to bypass on day one in case they can login from a mobile device, for example, but not from a work computer. It's the old registrations that may pose a problem and I'd guess 99% of questionable old accounts have 0 posts.

[Lee G]

Where do I find the logs? I don't see a log file anywhere obvious.

Also, aren't the Google IPs inherently whitelisted or do I have to go in and add them?

Sorry for the dumb questions, just trying to get this thing set up to a modest level of security without a lot of time I have to spend researching or editing.

In your admin cp

Go to Statistics & Logs

And you will see the bb logs there vB Bad Behavior Logs

[carsafety]

In your admin cp

Go to Statistics & Logs

And you will see the bb logs there vB Bad Behavior Logs

Duh! Thanks!

5300 entries in under 2 days.

Many are from User-Agent: Mozilla/5.0 (compatible; AMZNKAssocBot/4.0)

I'm guessing this one might be legit and part of my Amazon Associates program. What is the correct syntax to white list this please?

[Lee G]

You can white list that user agent
In the white list file, you will see how to do it

Using the user agent you posted, add this

useragent[] = "Mozilla/5.0 (compatible; AMZNKAssocBot/4.0)"

To the white list after

; User agents are matched by exact match only.

[useragent]

[carsafety]

You can white list that user agent
In the white list file, you will see how to do it

Using the user agent you posted, add this

useragent[] = "Mozilla/5.0 (compatible; AMZNKAssocBot/4.0)"

To the white list after

; User agents are matched by exact match only.

[useragent]

Thanks!

Couldn't find the whitelist file before as I was looking in the bad behavor folder, but I found it in the parent directory.

I think I got the range of IPs whitelisted too and they seem authentic as they are in Seattle.

I like how some user agents are disguised as Google Bot, even though they appear not to be Google at all.

[Lee G]

You will be surprised how many fake google bots are about
Some seo software runs a google view of things and also pretend to be google
Great way of finding your competition sniffing round your site

Something to watch out for and this happend to me last week

Some agents get blocked
I got hit about 1000 times in under an hour
All the guy did was set his scraper to rotate through a lot of agents
It was one of the most elaborate hits I have seen so far
The guy was still hitting while I added his ip to my htaccess and then the fire wall

I have also found that Project Honeypot gets busy and sometimes you might not get an ip blocked thats on their black list

I feel guilty with my war against the morons that hit me all day long with xrummer
All that money they have spent on it, we should happily let them have all our content

[Eric]

Thanks Lee and Alfa for helping out with answering questions, etc. My mind has been occupied by college and trying to find a job. Unemployment is rough so my only income right now is any online work I can find. I am trying to support my mods as much as I can, but hope folks understand other things may be a higher priority. :)

[Alfa1]

Thanks Lee and Alfa for helping out with answering questions, etc. My mind has been occupied by college and trying to find a job. Unemployment is rough so my only income right now is any online work I can find. I am trying to support my mods as much as I can, but hope folks understand other things may be a higher priority. :)
I have not received a quote on a job that I sent through your 'get a quote' link in your signature. Please check.

[Eric]

I have not received a quote on a job that I sent through your 'get a quote' link in your signature. Please check.
Would you mind re-sending it? I do not recall ever getting an email from you, and I cannot seem to find it anywhere. My apologies.

[Alfa1]

Will do.

Uhm, wtf? Im getting an extremely insane 404 page.

[Simon Lloyd]

Will do.

Uhm, wtf? Im getting an extremely insane 404 page.Lol, you want this one http://www.secondversion.com/project-quote/

[Strever]

installed

[carsafety]

I haven't had a single entry in my bad behavior log since Tuesday morning when I whitelisted the Amazon associates bot.

I double checked the syntax of the whitelist.ini file and it seems OK. Is it typical to go for a couple days with no rejections logged? I did verify that logging was set to YES.

[Lee G]

Noticed a fault with this
People with no user agent get the 0000000 error and then go on to rip the hell out the forums
Its not blocking those that try to sneak by with no user agent.
One person hit me several hundred times through this

Eric, if and when you read this, I am waiting on a quote from you for making an alteration to this for me, so if someone hits with a banned user agent it blocks the ip for x amount of days.
The clowns with xrummer hit multiple times from, rotating their user agents. So you might block one agent but they will get you with the others they have set up

Im almost on first name terms with the guys that hit me five or six hundred times a day

[Lee G]

If anyone else is getting the same as me.
I believe its one file to edit

Go into the Bad Behavior folder and find the responses.inc file
Open that with wordpad etc and look for

'00000000' => array('response' =>

Its about five lines down from the top

I have changed mine to

'00000000' => array('response' => 403, 'explanation' => 'An invalid request was received from your browser. This may be caused by a malfunctioning proxy server or browser privacy software.', 'log' => 'Required header \'Accept\' missing'),

I was tempted to change it to this

'00000000' => array('response' => 403, 'explanation' => 'You do not have permission to access this server.', 'log' => 'I know you and I don\'t like you, dirty spammer.'),

[Simon Lloyd]

Lee are you sure on this?, doesn't the 00000000 also refer to valid users?, users that do not have a user agent are being blocked by my version of vbBad Behaviour. Maybe ask Eric when he has time to take a look at your setup/settings.

[Lee G]

I just tried my forum from the user agent test page at bots vs browsers
http://www.botsvsbrowsers.com/SimulateUserAgent.asp

All worked fine on there som Im not blocking genuine users

Im not sure the above is working though. It was a 2am special this end

[Simon Lloyd]

.........Eric:
I have 286.000 entries in my logs. Its getting difficult to browse through this. Are entries really deleted from the log after 7 days or did BB block 286.000 attempts in the last week?

It would be very helpful to get an overview of registered members that were blocked. BB enables me to notify my members of troyans and other malware on their PC. So that after the removal of this malware they can freely browse my site. But there is no way that I can find time to review 286.000 entries per week.

Especially the entries that contain vb_login_username:

The IP whois does not work. It links to a white screen. Consider to use another whois service like http://whois.domaintools.com/Hi Eric, i have the same issue with the Cron job, it's not pruning, as for the WhoIs i constantly get a 404, you can have permission to use the WhoIs script contained in my mod here https://vborg.vbsupport.ru/showthread.php?t=264283 and incorporate it entirely within vB :)

[carsafety]

Thanks!

I've just enabled this mod to test it. No issues with ForumRunner so far, at least on the iPhone app. Apparently, Tapatalk does something different inherently that needs whitelisted.

Two questions for the devs or anyone who knows:

1) If I set "Bypass Users" to "Yes" and "Join Date" to 0, does that mean it will bypass all registered users?

2) For "Operating Mode", is normal mode "Yes" or is strict mode "Yes"?

I'll report on traffic and affiliate revenue in a week or so.

Week to week, affiliate revenue was about the same. Most other stats were similar or down slightly. Overall data transfer was down 15% or so, presumably a benefit of bad behavior.



I haven't had a single entry in my bad behavior log since Tuesday morning when I whitelisted the Amazon associates bot.

I double checked the syntax of the whitelist.ini file and it seems OK. Is it typical to go for a couple days with no rejections logged? I did verify that logging was set to YES.

I am still having this issue. I uninstalled and reinstalled from scratch and it began logging again, about 10 entries in a minute or two. Most were Amazon affiliate bot. A couple were forum runner that I hadn't noted before.

I again edited the whitelist.ini and it no longer logs anything. I can't believe no other bots are coming to the site. I did notice that when I first opened the whitelist file it appeared in notepad without line breaks. After the edit, it appeared with line breaks where I would have expected to see them. Other than that, it seems correct. I'm not sure why else editing the whitelist file would cause logging to stop. I'm guessing it is still rejecting evil clients, because my data transfer stats were consistently 10-20% lower.

Sorry for being such a noob. I just can't figure out why it has this behavior. It's probably something obvious I'm just not seeing it. My whitelist file is attached in case it's a simple case of syntax somewhere maybe someone will spot the error.

Is it needed to whitelist it? Is BB blocking anything in Forum Runner?

I added forumrunner URLs as listed to be safe. The one that showed up in my logs was Forum runner for Android at the request.php URL.

[carsafety]

Sorry for being such a noob. I just can't figure out why it has this behavior. It's probably something obvious I'm just not seeing it. My whitelist file is attached in case it's a simple case of syntax somewhere maybe someone will spot the error.


Problem solved. Helps to read the text indicating the IP addresses have a special format and not just wing it...

[carsafety]

Does anyone have a working whitelist entry for hostracker or hyperspin? I see an apparent hostracker bot rejection, but can't really verify if it is legit.

[Lee G]

Host tracker operate out of The Planet
I dont know anything about them, but their ip is listed as 67.18.217.22

http://www.ip-adress.com/reverse_ip/host-tracker.com

http://whois.domaintools.com/67.18.217.22

Hyperspin also work out of The Planet

http://www.hyperspin.com/en/

Ip 174.132.46.210

http://www.ip-adress.com/reverse_ip/hyperspin.com

http://whois.domaintools.com/174.132.46.210

I dont use either of the services, so have no idea of the user agents :o

[carsafety]

That should be correct.


Normal mode should be set to yes, not strict mode.

I think Normal mode might be "No"? I had a couple members rejected after I switched it to Yes. At least one was at work and probably through some kind of proxy.


Host tracker operate out of The Planet
I dont know anything about them, but their ip is listed as 67.18.217.22

http://www.ip-adress.com/reverse_ip/host-tracker.com

http://whois.domaintools.com/67.18.217.22

Hyperspin also work out of The Planet

http://www.hyperspin.com/en/

Ip 174.132.46.210

http://www.ip-adress.com/reverse_ip/hyperspin.com

http://whois.domaintools.com/174.132.46.210

I dont use either of the services, so have no idea of the user agents :o


Thanks! I tried to find it again and with all the entries in the log I didn't have time yet to sort through them all another time and figured someone had something known to work.

[Alfa1]

I think Normal mode might be "No"? I had a couple members rejected after I switched it to Yes. At least one was at work and probably through some kind of proxy.
Operating mode should be set to Normal mode. The radio buttons are confusing as these do not indicate what is normal mode and what is strict mode. You should indeed set it to 'No'.

[alaska_av8r]

Guys pardon this if this is a stupid question, but I am running bad behavior and having problems with my forums slowing way down, I haven't determined the cause of this but did find this in my server error logs, is this someone trying to hack my forums??

[Tue Jun 14 19:18:11 2011] [error] [client 71.199.107.79] File does not exist: /home/marin49/public_html/modules, referer: http://www.boatinghowto.com/ [Tue Jun 14 09:40:23 2011] [error] [client 71.199.107.79] File does not exist: /home/marin49/public_html/modules, referer: http://www.boatinghowto.com/ [Tue Jun 14 05:57:55 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/admn [Tue Jun 14 05:57:55 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/admm [Tue Jun 14 05:57:55 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/databaseadmin [Tue Jun 14 05:57:54 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/mysql-admin [Tue Jun 14 05:57:54 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/mysqladmin [Tue Jun 14 05:57:54 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/webdb [Tue Jun 14 05:57:53 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/websql [Tue Jun 14 05:57:53 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/sqlweb [Tue Jun 14 05:57:53 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/webadmin [Tue Jun 14 05:57:53 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpmy-admin [Tue Jun 14 05:57:52 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/php-myadmin [Tue Jun 14 05:57:52 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpmanager [Tue Jun 14 05:57:52 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/pma2005 [Tue Jun 14 05:57:51 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/PMA2005 [Tue Jun 14 05:57:51 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/p [Tue Jun 14 05:57:51 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/mysqlmanager [Tue Jun 14 05:57:50 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/sqlmanager [Tue Jun 14 05:57:50 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.8.2 [Tue Jun 14 05:57:50 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.8.1 [Tue Jun 14 05:57:49 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.8.1-rc1 [Tue Jun 14 05:57:49 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.8.0.4 [Tue Jun 14 05:57:49 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.8.0.3 [Tue Jun 14 05:57:48 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.8.0.2 [Tue Jun 14 05:57:48 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.8.0.1 [Tue Jun 14 05:57:48 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.8.0 [Tue Jun 14 05:57:48 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.8.0-rc2 [Tue Jun 14 05:57:47 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.8.0-rc1 [Tue Jun 14 05:57:47 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.8.0-beta1 [Tue Jun 14 05:57:47 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.7.0 [Tue Jun 14 05:57:46 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.7.0-pl2 [Tue Jun 14 05:57:46 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.7.0-pl1 [Tue Jun 14 05:57:46 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.7.0-rc1 [Tue Jun 14 05:57:45 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.7.0-beta1 [Tue Jun 14 05:57:45 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.6.4 [Tue Jun 14 05:57:45 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.6.4-pl4 [Tue Jun 14 05:57:44 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.6.4-pl3 [Tue Jun 14 05:57:44 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.6.4-pl2 [Tue Jun 14 05:57:44 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.6.4-pl1 [Tue Jun 14 05:57:43 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.6.4-rc1 [Tue Jun 14 05:57:43 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.6.3-pl1 [Tue Jun 14 05:57:43 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.6.3 [Tue Jun 14 05:57:42 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.6.3-rc1 [Tue Jun 14 05:57:42 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.6.3 [Tue Jun 14 05:57:42 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.6.2-pl1 [Tue Jun 14 05:57:42 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.6.2 [Tue Jun 14 05:57:41 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.6.2-rc1 [Tue Jun 14 05:57:41 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.6.2-beta1 [Tue Jun 14 05:57:41 2011] [error] [client 85.114.137.170] File does not exist: /home/marin49/public_html/phpMyAdmin-2.6.2-rc1

[Simon Lloyd]

Certainly looks like someones trying to find a back door, get in touch with your hosts to help resolve this!

[Alfa1]

Start by blocking that IP and only allowing whitelisted IPs in phpmyadmin and admin panels.

[Lee G]

Looks like they are running the same idiots guide for dumb hackers disk someone tried hitting me with from Taiwan the other day. ip 140.109.65.35

Block the ip and make sure that both your mod cp and admin cp folders are password protected
Or you set those folders to only be accessible from certain ips if you and your mods are on static ips with a simple htaccess file

[alaska_av8r]

If I password protect mod cp and admin cp folders, how will this affect vbulletin and will it affect upgrading etc when I go to upload the new files?

Kind of new to some of this sorry...

[Simon Lloyd]

Nope won't affect them at all :), so i assume you're runing cpanel? if you are then its dead simple to do and you can assign each of your mods and admin seperate usernames and passwords to access the folders.

[Lee G]

To secure your htaccess file, to make it unreadable by anyone, add the following bit of code to it

<Files .htaccess>
deny from all
</Files>

If they cant read it or access it, they cant hack it or find what ips are banned

By password protecting your admin and mod folders.
If anyone was to hack your password, they can delete the odd member or post on the forums front end, if they get to the back end, your screwed

I was one of the many victims of the base 64 code injection hacking that went on last year
Hindsight is a wonderful thing, I wish I had known about doing the above at the time

[alaska_av8r]

thanks so much for all your helpful hints and I will do all of the above mentioned steps, being new to this stuff it really helps to have folks around with the know how....

tim

[alaska_av8r]

I don't mean to hijack this thread, but with this .htaccess file, exactly where do i put it...I do have cpanel and I put it on the same level as public_html (i am assuming thats called root directory) and I put a copy of it in the public_html directory. Do I only need one and exactly where do i put it.

thanks

[Simon Lloyd]

To keep this thread clean i've pm'd alaska_av8r

[Eric]

Sorry guys, I have a lot going on atm. Not only college, but now a... well, won't get into that. But I will try to get back to this as soon as I can.

[viper357]

That should be correct.


Normal mode should be set to yes, not strict mode.

Operating mode should be set to Normal mode. The radio buttons are confusing as these do not indicate what is normal mode and what is strict mode. You should indeed set it to 'No'.
I'm confused, lol, should the Operating Mode be set to Yes or No? Thanks.

[carsafety]

I'm confused, lol, should the Operating Mode be set to Yes or No? Thanks.

It is your choice. "Yes" is more secure but also has the potential to block a few legit clients who are using networks at work that have certain firewall or proxy setups.

[Alfa1]

I'm confused, lol, should the Operating Mode be set to Yes or No? Thanks.
Set it to No.

[viper357]

Thanks guys.

[viper357]

I'm seeing a lot of these in the logs, does this mean that it is blocking blackberry users?

GET /showthread.php?p=470371#post470371/external.php?type=RSS2&forumids=219 HTTP/1.0
Accept: */*
Connection: keep-alive
Host: www.marineaquariumsa.com
Cache-Control: max-age=259200
Via: BISB_3.5.1.71, 1.1 pmds109.bisb2.blackberry:3128 (squid/2.7.STABLE7)

[Simon Lloyd]

Thats just someone picking up your RSS feeds via their blackberry, or it's a bot with a malformed useragent attemptng to retrieve your RSS of your site.

[viper357]

Thats just someone picking up your RSS feeds via their blackberryThat's not cool, it's reducing traffic to my forum from probably legitimate members that are just not logged in, and now it seems to be blocking Amazon and Alexa, how do I unblock them?

ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)

[viper357]

Actually, nevermind, I'm going to uninstall this mod, seems like far too many legitimate users and websites being blocked. My site is too busy to have to keep checking logs like this. Excellent mod though but just not for me, thanks.:)

[Alfa1]

You do know that alexa is spyware and that amazon is a cloud? Its completely unrelated to websites like amazon.com Users operating from a cloud surely are no legitimate users. This addon does not block websites, but bots.

If you want to unblock a user agent, then just add it to the whitelist.ini file.

[viper357]

How can Alexa be spyware, it's a website/company just recently purchased by Amazon, surely they wouldn't buy spyware companies? Alexa is well known for it's site monitoring statistics, I use it all the time, to me it's a simplified version of Google Analytics as it crawls your website in a fairly similar way. Or is there something I am missing?
If you want to unblock a user agent, then just add it to the whitelist.ini file.While I can appreciate that 100%, I really can't be spending hours and hours going through literally hundreds of log entries every day looking for legitimate user agents.

Like I said, this seems like a really fantastic mod, but it's just not for me, my forum is really busy and I need to concentrate on the content and the members rather than digging through log entries on a daily basis, but thanks for your help, as always it is appreciated.:)

[Lee G]

Alexa is about as much use as a chocolate tea pot.
A lot / if not all of their data is from people that have their tool bar
About 1/1000 of my traffic uses their tool bar.
Compare that to the google tool bar users and there is no comparison on which company you trust for true statistics

[Simon Lloyd]

@Viper357 look here and whitelist the useragents there http://www.useragentstring.com/pages/ia_archiver/