DNSBL/Open Proxy-Blocking
 

History

I've had some problems with abuse via open proxies for a time ago, and when we were banning abusers, they always found a new proxy to use and register new accounts with. Since this forum was a large type we could'nt just ban e-mails etc just like that, because this was leading to a very large amount of other banned users too.

At IRC, in the other hand, we had Open Proxy Monitors, that was banning everything that was blacklisted in some DNSBL-databases. No spammers had a chance to get in there as long they were listed in such database.

This is a plugin that blocks blacklisted hosts from some different DNSBL's. It uses the global_start-hook, a very simple handler for blocking proxies, and a vBphrase called OPM_Deny.


April 2006

The source has been rewritten a bit. The proxychecker is now using a cache that, by default, stores all ip's in a database for 6 hours. It scans some DNSBL's and can be configured to block proxies from bitmasks (defined in the plugin) which makes it a little bit more reliable, because it does'nt block everything it see).

Configuration is made from the plugin (hopefully there will be a nice admin interface in the future). Exceptions (ip's that can pass through this system even if it is a proxy) are also handled differently now.

// CHANGES
//
// 2008-09-20 (2.0.8)
//
// * Changed the routines for how to handle inclusion/exclusions
// * Splitted up plugins for 3.5/3.6 and 3.7
//
// 2007-08-05
//
// * Fixed reported bug, based on resolved hosts ending with 127
// * Changed database-tables to get rid of (hopefully) duplicate keys
// * Added resolver-function
// * Added two new block-methods available at the efnet-rbl
//
// 2006-06-28 (2.0.6/Another fix)
//
// * Proxyinclusions/exclusions didn't work properly
//
// 2006-06-28 (2.0.5/Fix only)
//
// * Fixed a bug in the $block-array that affected some of the blocking results
//
// 2006-06-28 (2.0.4)
//
// * opm.tornevall.org has a new entry for anonymizers, added support for this
// * Default value on "block everything detected" in plugin changed to "no"
//
// 2006-06-26 (2.0.3)
//
// * Created options for admincp (removed plugin-configuration)
// * Fixed a bit-bug for njabl
// * Plugin is now a function (rbl_livecheck) for external lookups
// * Added options for "only block on newuser-registrations"
//
// 2006-06-22 (2.0.3 RC)
//
// * The monitor is now a function
// * Added small compatibility with other plugins (with return)
//
// 2006-05-13
//
// * sorbs zones added (no bitmasking)
// * opm.blitzed.org removed
// * time() changed to TIMENOW
//
// 2006-04-21
// ==========
//
// * proxyinclusions
// quickly add own hosts that should be treated as a proxy
//


How does it work with other vBulletins?

This filter actually works with both 3.5 and 3.6, but for now, they will be separate versions, but for 3.5 and 3.6 you should look here and for 3.7 you should look here.


How to use the compatibility thing

If you have a plugin that you want to use together with the proxy monitor (only returns a value if a an ip-address is registered as a proxy or not) you can call the function rbl_livecheck like this (example):

Report bugs if you find them...



Don't forget to install it :)

What happened to the xml?

There it was. It disappeared when I was editing this thread. :)

You know you can make this a product and have it install the phrase auotmatically on install right?

I did'nt know that. I know very little about plugins. Is there any instructions for me somewhere about it? :)

Here you go. ;)

my feedback: actually, your script is overloading the servers on both sides as it connect each time a page is loaded... on irc there is no problem as you connect only once per logon, but on web, each time a page is loaded the script is activating...

what would be best is to cache the data received from the OPM databases, and make a verification only when a new session is built... this would ensecure the connection, and spare some contacts with the dB!

I don`t tried to install this plugin yet and the post about overloading server stoped me from installing but still iI think that this is a great idea and you should work on it because we all sooner or later will have problems with users that use some proxy servers and feel that we can`t do anything with them.

Yes, that's a good idea, but that's something that I probably can't fix on my own, due to a lack of necessary php-skills. :)

we all can help on this... you started a good thread/hack, i know that some others will be able to add to it... The Boofoon guy already had!

this is quite simply btw... a cronjob to query the proper servers will store the datas in the datastore, and we only retreive that information when needed... the cronjob can be set to update itself once a week or so, because these servers are not updating the listings that often...

This is an awesome idea....I think nexialys has an excellent point and that will make it an invaluable product.

Very nice a idea; thanks a huge bit for starting this, hopefully anyone with sufficient php skills will be able to "finish" it :)

cant it be placed at the start of session?

that's what i suggested... ;)

I use the PEAR Cache library for some of my more esoteric hacks, happy to write a PEAR compatible version if people feel that PEAR is an acceptable pre-requisite.

Of course... if someone can write a datastore version it would probably be better :)

this would be better in the fact that we need these addons to use the vBulletin protocol/codes ... or why would we code a software that big if nobody use it.. :)

Yah, this is very cool b/c we need this... we have some abusers that like open proxys.

As to the pear suggestion, hmm, would that be a stopper for some folks? Is it available on all hosts using PHP?

I might be able to be helpful with cron and cacheing things in the file system or database. I'll be back sometime after jelsoft does their release to work look at your code. For the moment, I'll subscribe to the thread.

I'm very happy that the plugin gave so many positive reactions. I look forward with the last "fix" for this one, because of my own lack of php-skills. Thanks a lot! :)

This idea rocks!

*subscribed*

Is anything happening with this hack? I think its a great idea if the speed and load issues could be worked out.

For those that are interested there is a nice php script that can be added to VB to block proxies. It will write to a mysql table and check the table first for banned addresses. For you coders maybe this script here can be reworked to add these features.

http://phprbl.init1.nl/

That's the way the resolver for opm.tornevall.org works. The only problem is the cache-part for me. Does the url help in a similar way?

It's still a question of querying that data only on session (1 time) instead of global (every page load)

There's no longer a session.php. It's in class_core.php I think
I browsed the new code and came away muttering to myself...but ima noob :ermm:

I would think the best thing to do would add an extra column in the session table to do a hash of the "host" column, so you know when it changes that you need to re-check the IP.

Then add another (bool) column to store if it's an open proxy or not.

Find a hook in the session management to add in the necessary check / storing code, and you should be good to go.

Then each time a page is requested all it has to do is a quick MD5 of the current IP against the stored IP, if it's the same and it's okay, then nothing new, if it changes then make a new RBL request and go from there.

Session is a HEAP table, so it will go fast, and also clear out over time.

I could knock this out in an afternoon (assuming there are hooks in place, otherwise it would require a couple lines of manual file editing).

Easy as pie.

Well TMM I install it just to try. The idea is super, but my knowhow is a little to low for helping you out.

This option should be standard in the boardsoft a lot of us have problems with abusers who are changing proxy's within the hour.

I aplaud you TMM

Ad

Well, since there has been no reply about my concept, I'll go ahead and code a session based one like I stated above.

Your concept is good. For my purposes though I found a better solution. I hacked a couple scripts to 1) check the IP against a banned table first if not found then check the rbl servers 2) if found in the rbl servers write the blocked IP's into my .htaccess file blocking the user completely 3) customized my 403 page to display the service they are blacklisted on and provide removal instructions.

Anyone wanting the script can PM me. It would be nice if someone that was a good coder (not me) went through it refined and optimized it.

Thank you for that... I have not much knowhow about this, so if you can do it then it's great.

Ad

I'm about 99% done. All I had to do actually was add a single column to the session table labeled "OPM".

The code is flexible enough that you can add more RBL's and choose the reponse code(s) you want to blacklist. (Since not all return the same 127.0.0.x code, or the last number can have different meanings).

Yes, I could of added an additional blacklist table, however I wanted to do this quick and easy, and also this is just a realtime blackhole, not a permanent one. Though your hack (webspider) does sound worthwhile. I actually have only had to block a couple users via my firewall that were severly abusing the forum.

Anyhow, once I get that last little kink worked out, I'll post the code in this thread.

Alright, I've finished the code... My last problem was that the templates were not showing up, the issue came because the global_start hook occurs before the header & footer templates are fetched, so I had to add them in manually.

There's 3 simple steps to implementing this.

1 DB addition
1 Phrase addition
1 XML file to import

I guess this could of been a "product" or whatever to automate the DB & Phrase, but I haven't had a chance to mess with that yet, and I don't plan on supporting this, so please don't ask.

The code is simple enough, I think a person could figure out how to add additional RBLs.

Anyhow, I'm not trying to step on the original author's toes with this hack, this is just a slightly different alternative that only checks (and saves) once per session (instead of every request).

Could someone remake a bit @eoc_Jason code into produck so it could be installed and uninstalled right from the ACP? Thank you :)

ps.
I think that this option when will be done and working on 100% should be included as a standard option.

I actually modifed it even more from what I posted for my forum, only to run the check on specific user groups.

I'll probably futz with it some more today, to make it an actual "product" for my site.

TMM-TT (or anybody) can use the file I uploaded and expand upon it however they want. I don't want to step on anybody's toes with this hack, which is why I'm offering for someone else to manage it. However if nobody else wants to step up to the plate I am more than willing to do it.

EDIT - Here is an update from the previous page. This is released as a "product" so that it adds the database field & phrase and everything automatically.

You have some typo errors at the bottom in the msg but thats no big deal.

Installed & tested it with my Hide IP platinum application and it blocks the proxies it uses that are blacklisted :)

I am extremely impressed. Well done eoc Jason you are a star.

I personally think this has to be the hack of the month!

Jason you earn a price :rolleyes:

Installed and tested with several proxy proggies and it works fine !!!

Many thanks from me and the mods of my forum.


Ad

PS:

Perhaps it's better to start a new thread or edit the first post (mister moderator) while things are different now then mentioned in the first post IMHO.

lol, I just noticed there was a type in the OPM_Deny phrase, it's pseudo engrish. Oh well. I'm sure people will customize it to meet their needs anyways.

Like I said, you can have it check other RBL's, and set only the returned error code that you want (if they return multiple), it will stop after the firs the blacklisted found. However I think spamhaus is enough since it is a combination of the 3 big ones out there.

You can additionally test it if you edit your session table directly and set your session OPM = 2. (I had a heck of a time finding an open proxy!).

I've found that about 1 or 2 out of the current users online is marked as an open proxy (I have about 250-400 people online at a time usually). Some are false positives though as I got a few complaints and when I checked them out, they were entries that haven't been tested in weeks or months!

I have several usergroups which people get auto-promoted, so only my guests, and lowest registered usergroup actually get checked, which I might even allow guests too, not sure at this moment in time.

If you only want to check certain usergroups, you can use the following IF statement around the two function calls.

Nice work! I had kludged something together earlier, but a plugin is a lot better. I've installed it and will test it out :)

I'm not sure I agree with the comments on caching. This plugin uses gethostbyname() to do a DNS request to the blocklist servers. This DNS request will be issued to the (colo provider's) DNS cache and that server will already cache the answer for you. This server will keep it in the cache as long as the TTL of the record in the blocklist, so you are caching it as long as the blocklist specifies it, which is often hours or days, and you have the best chance of having an up to date result, because your DNS takes care of caching and retrieving.

Caching would also make the code more complex and prone to errors, also it costs extra time for a feature that DNS already provides. A network DNS lookup is also often faster than a database table lookup and does not stress the disk.

When you check one blocklist, the network load is comparable to doing HostnameLookups On in Apache, which is minimal for all but the largest sites. If needed, I would advise checking a composite blocklist that mirrors many other lists, so you only spend one DNS packet in and out.

eoc_Jason

Doesnt look like anyone wants to step up... Anyway, great product and its already working and has stopped a few of my problem childern on my site. Though my only conplaint was it blocked one of my regulars who isnt even on a open proxy BUT who's ISP has lots of blacklisted IPs because of spam/proxy abuses. He is using a static ip so he is having to submit it to spamhaus and see if they will remove it. Plus he is contacting his ISP about the issue...

As for any future updates you or anyone might make on this product it would be nice to have a exculed IP list via your admincp. Basicly, a nice little field were I could add ips that should be exempt from the blacklist access block even if they are blacklisted via spamhaus or some other list you might be using via this product. But then again one user having to deal with this is all that big of deal....

Anyway, thanx again for a great product.

AWESOME!! I have this one idiot that keeps coming back saying crap like "u can't get rid of me" blah blah. I hope this does, at least it is a step in the right direction.

Skydancer - I agree with your statements about how DNS "should" cache the info, however, not knowing what all RBLs set their TTL to, or also any mis-configurations, having one little bit of info stored locally makes sense (to me).

The current RBL lookup in the code is spamhaus' list, which is actually a combined open proxy list from 3 sources. So you get a 3-fer-1 in the query.

Lastly, the session info for vB is a HEAP table (meaning it's only stored in memory, not on the HD). vB does a "SELECT *" from the table when doing it's inital session stuff, so there is no added overhead in that case.

Like I said in a previous post, the worst-case scenario that I can think of is if the RBL goes does or does not respond in a timely fashion. I did not put in any timeout / exception handing.


cnutter - Yes, I had the same issue with a couple of my members getting dumped with bad dynamic IPs. However, all the sites that spamhaus uses has extremely simple (one-click removal) links, which will remove the IP and by the next update (hourly) should no longer be blocked. An IP exclusion would not work for most people as they have dynamic IPs. Instead you might want to consider doing it by usergroups, as I posted a snip of code above about bypassing the check for certain groups.

Regarding the "Stepping on someones toes"; it's ok with me if you step on my toes, because I needed help with this plugin from the start myself. I knew that you guys in here could help me with this one, to make it better. So any ideas are still welcome. My RBL-DNS opm.tornevall.org is free to use as long as it exists, and will be continuously updated with new proxies, and old proxies will be marked as "not working" when they are not reachable anymore.

The values for working and not working are:

.. and the blitzed-one is used if the proxies are matched with the blitzed database (Currently around 4200 out of 28300 registered proxies has been matched to their database).

By the way, thanks for all help. :)