Administrative and Maintenance Tools - vbStopForumSpam - known spammer lookup for new registrations

vbStopForumSpam

This provides access to a RBL type system for forum admins, listing known spam IP / email / usernames. The RBL database is provided by www.stopforumspam.com. You do NOT need an API key from the website in order to access the database. only to submit data if you should wish to do so.

At the point of user registration, the mod checks if the IP number / provided username / email addresses appear on a block list and can block the registration.

Whilst this isnt the most perfect way to stop all forum spam, its another step that spammers have to overcome.

VB4 here https://vborg.vbsupport.ru/showthrea...hreadid=230921
Its the same code, it works in 3.54 to 4.0


What it does

It checks with a remote database of known forum spammers. Their IP number, email address and forum username are tested and based on your configuration, you can reject / log / accept user registrations based on what you get back.

This version doesnt have
- whitelisting or the ability to submit users to the database but it will within the next week.
- automatic user deletion / post / PM purging. There are good tools out there already, this does something else.

Instructions are included in the installation.txt file - PLEASE read it first and dont forget to actually upload the files in the upload folder, otherwise it WILL kill your registration progress and you wont see the log file options in admincp. You do not need to download the product-vbstopforumspam-3.54.xml file unless you are using a vBulletin version older than 3.6.0

Changes to vB
- 3 new database tables
- 2 database table alternations
- No new templates.
- 2 Hook (register_addmember_process & register_addmember_complete)

Ive tested it but had feedback that it works with versions as old as 3.6.2... Support should go back to older versions, as long as they have hook support for register_addmember_process / register_addmember_complete

Known to work - tested by me
- vBulletin 3.6.8 on Apache 2.2 / PHP 5.1.2 on Linux using cUrl
- vBulletin 3.7 Gold on Apache 2.0 / PHP 4.4.3 on Windows without cUrl (template changes wont work on 3.7 - thats in the next version with auto template changes)

For code to submit spammers to the database, check this post for code changes
https://vborg.vbsupport.ru/showpost....&postcount=288

Reported in the thread to work
- 3.6.1, 3.6.2, 3.6.9, 3.6.10, 3.7.0, 3.7.1, 3.7.2, 3.7.3, 3.74, 3.80, 3.81, 3.82, 3.83, 3.8.4, 4.0beta3

If you have 3.54, then you can use the product-vbstopforumspam-3.54.xml file attached instead of the one in the ZIP file, which will allow older vBulletin versions to access this mods' features. I personally havent tested this version, its a user contribution, thanks to Darrell Mobley, that changes the way the XML works when imported into older versions.

Installers should remember to refresh their ACP navigation window when they first install it so they can see the new log file menu item.

REQUIRES MySQL 4.1.1+

Future versions
- Automatic integration into vBulletin to add users to the stopForumSpam.com database from a form
- Whitelisting of username / IP / email addresses
- AJAX integration to allow for lookups from within the users profile
- Decreased remote query count from three per user to one per user.

Versions / Changes

0.1 Initial Release

0.2 pedigrees special brew birthday release.
- Small security update. If you have 0.1 installed, download 0.2 and replace your existing functions_vbsfs.php with the one in the archive. It just tests to see if its running inside the VB framework before anything else. This is what happens when you code at 2am after drinking wine

0.3
- stopped it processing valid registrations twice
- moved all non-function code into the plugin. Not a big one as 0.2 basically did that
- fixed a typo in the log pruner that stopped it working (404)
- removed unused fields from the database for people with mysql that doesnt support varchar > 255 (ie mySQL4). If you have 0.2 installed and dont need to prune your logs just yet, you dont really need to install this version but can instead wait for 1.0 unless of a massive security update.

0.4
- logs registrations that arent/wouldnt be blocked
- fixed XML errors when username has a space it in
- tightened up the cache so that it doesnt test a username against an email name to give a bypass result (for when a username is an email address that isnt banned where the email address is)
- fixed some basic logic errors in the PHP

0.6
- Should work on PHP 4.4 now - rewrote the XML with PHP4 in mind (tested on Apache2.0/PHP 4.4.3)
- Fixed a caching system where data wasnt being updated correctly which could cause a remote query when one wasnt needed
- Possible false negative situation when a spammer was blocked due to SFS.com being down who then visited again when it was up but within the cache expiry time
- Remote query failure when the result page isnt XML should work a bit better now. It does a very basic test for valid XML results.
- Fixed log purging (again) and it should actually work properly now.
- No longer requires PHP5
- The log viewer now links to a user profile when registration is allowed.

v0.61 - Removed a template change that was invalid vBulletin code. The package you download will still say its 0.60 however

NB : When upgrading from any version to 0.6, you must remove and then add the plugin due to changes in one of the database tables

You need to have an API key from www.stopforumspam.com in order to submit data, its free and easy to get... You DONT need an API key in order to use this mod however, only to submit spammer data.

Issues are
- The usergroup permissions / view details etc DONT work. I jumped the gun and put the permissions controls in there before I put the code in. Please delete the includes/xml/bitfield_vbstopforumspam.xml file and rebuild your postbit

Installation
- Follow the instructions in the zip file, that includes upload the correct folders
- ONLY download the 3.54 xml file if youre using a vbulletin version prior to 3.6.0. use this file to install the mod instead of the xml file in the zip file.

Please click Installed

[pedigree]

ok, your webserver is hosted at iWeb. The server is failing to resolve the IP to the domain. There really isnt anything that you can do to hurry it up other than to ring them and give them a "oh, DNS isnt working"

Sorry

Most DNS can take up to 24 hours to propagate, depending on the DNS servers configurations.

[Maxima]

Originally Posted by pedigree

ok, your webserver is hosted at iWeb. The server is failing to resolve the IP to the domain. There really isnt anything that you can do to hurry it up other than to ring them and give them a "oh, DNS isnt working"

Sorry

Most DNS can take up to 24 hours to propagate, depending on the DNS servers configurations.

I sure will ring them up!

Thanks!

[SirShaw]

great mod pedigree, certainly stopped all the ahole bots from signing up and trolling my board. I was getting like 10 a day! A++

[kyle1745]

Here are the results:

Server: 208.116.30.21
Address: 208.116.30.21#53

** server can't find www.stopforumspam.com: SERVFAIL
Server: 208.116.30.21
Address: 208.116.30.21#53

** server can't find www.stopforumspam.com: SERVFAIL


; <<>> DiG 9.2.4 <<>> www.stopforumspam.com
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 64946
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.stopforumspam.com. IN A

;; Query time: 92 msec
;; SERVER: 208.116.30.21#53(208.116.30.21)
;; WHEN: Thu Feb 4 18:03:30 2010
;; MSG SIZE rcvd: 39

[pedigree]

Im hoping that some of these dns failures are fixing themselves now.

[websissy]

Not one bot registration got past my junk yard dogs, vBSFS and STRB in the past 10 days. Legit users got in without issues. But between them vBSFS and Stop The Registration Bots stopped every single bot.

Way to kick butt, pedigree. {loud applause, cheering, foot stomping, howling and wolf whistles} I say, HELL YES!!!

Thank You!!! :up: :up: :up: :up: :up: :up: :up: :up: :up: :up:

[burlesque]

This wasn't working on my site for the last couple of days and as a result nobody could register. I turned it off three hours ago and since have had 20 spam registrations.

Luckily the DNS now seems to be working again and so I've turned it back on again.

This little outage has made me doubly appreciate just how vital this mod is to ANYBODY running vbulletin. If you have to install just one mod, make it this one!

[skippybosco]

@burlesque: While I don't suspect SFS will be moving DNS again soon, to future proof yourself, be sure to set the "on network connection issue" in the SFS Mod settings to allow registration instead of rejecting.

[djbaxter]

Originally Posted by skippybosco

@burlesque: While I don't suspect SFS will be moving DNS again soon, to future proof yourself, be sure to set the "on network connection issue" in the SFS Mod settings to allow registration instead of rejecting.

That's actually a good idea.

[websissy]

@skippybosco: Yes, I agree with djbaxter this IS a good idea. That's why I already had my installation configured that way.

I also increased my local cache retention time from 30 minutes to 480 minutes (8 hours) and I increased my local log retention time from 30 to 60 days. increasing the cache time dramatically reduces the number of lookup requests I send to the central server and greatly reduces the likelihood I'm going to find that server too busy to handle my requests. At the moment, my server processes and rejects about 120 bot registrations a day (that's 5 per hour). By increasing my cache time from 30 minutes to 8 hours, I'm retaining just 40 records in my recent registration attempts cache rather than 3; but I've reduced my load on the central server to roughly 1/13th of what it would have been otherwise.

In short, from what I can tell by increasing my cache time to 16 times as long, I've reduced the load my site places on the central server over 92% and I've also improved the performance of SFS on my site because its getting a local cache hit in many more cases rather than waiting for 3 database queries to occur on an overloaded remote server. Frankly, I recommend those changes to EVERY site that's using SFS. I suspect we'd totally eliminate the central server performance issues if we did this.


@pedigree:

I'm confused about something here. It occurred to me this evening that I never actually saw anything that said this addon checks its own local database of rejected registrations using the registering user's email address, IP address and username BEFORE it goes off to the central server to check the database there. Yes, I realize it looks in the local cache covering the last xx (user configurable) minutes of registration attempts before it goes to check the host, but since the "standard" user-configurable cache time is set at 30 minutes whereas the local rejected user registrations table involves weeks or months of rejected registration history (I'm keeping 60 days of history) it seems to me the load on the central SFS server could be HUGELY reduced by increasing the cache time to 4 or 8 hours and then checking BOTH the cache from the last (user configurable) minutes and the local rejection log database for the last (user configurable) days rather than going off to check the central database practically everytime a bot tries to register on any site.

For example, my site receives about 120 bot registration attempts per day. That's 5 per hour or 2.5 bot registration attempts every 30 minutes. Compared to the 1,200 rejected bot registrations captured in our local SFS rejection log in the past 10 days, that suggests the 30 minute local cache is so small it's almost useless.

As I examine my own local SFS rejection log which already contains 1200 bounced registration attempts after just 10 days, I can see many of these bots come back time after time every day and try registering with the same IP address, username and email address over and over again. Furthermore, many of the bot registration attempts occur day after day several times per day and then the bot goes away and comes back again after 24 hours or so. With those behavior patterns in mind, it looks to me as if the load on your central server could be cut WAY down to maybe 5% or 10% of the current load if the hack was just modified to first check the local cache, then query the local rejection log for the last 5 or 10 or 15 or 30 (user configurable) days BEFORE going off to ask the remote server if this registrant has been reported as a spammer by any other site. If you combined this mod with a signficant increase in the cache time it looks to me like you'd eliminate most of the query requests the central server now sees -- especially on sites that have been around a while.

It's just a suggestion, pedigree, but I suspect if you'd just increase the cache time and make this one simple change to your look-up logic your problems with database and name server performance at the central server will completely disappear. As it stands now, with at least 1000 sites using your product and accessing your database and an average of lets say 120 bot registration attempts per day per site (that's my own site's average), that says at least 5,000 bot registration attempts per hour (that's up to 15,000 central database queries per hour) are being handled by your product worldwide. However, since recording the fact that they're using your software is NOT mandatory here for ANY webmaster, I'll bet that 1000 site estimate is low and the actual number is 2 or 3 times that. Even if we assume half those 5,000 bot registration attempts are never reaching your central database because the local cache is blocking them (with a 30 minute cache time I'll bet the percentage being blocked locally is much lower than 50%), that still means your local database receives up to 7,500 query requests per hour to look up the IP address, email address or username of a bot who in all probability has visited the requesting site one or more times in the past few hours, days or weeks.

If my "SWAG" is right and there are actually 2,000 or 3,000 sites using your software rather than the 1,000 sites shown as having clicked "install" here at vbulletin.org, then your central server could be receiving 15,000 - 22,500 query requests per hour. That starts to sound like a helluva LOT of database work and would certainly explain why the central server is overloaded. To make matters worse, if my guess about the local cache hit rate is correct, then you're getting a much smaller percentage of local cache hits than the 50% I assumed above. In that case, your central server's database workload could be as high as 30,000 to 67,500 query requests per hour rather than 7,500, 15,000 or 22,500.

But the good news is, if my hunch about the cache time being too short is correct, you could reduce that query load by 93% just by increasing the cache time from 30 minutes to 8 hours. That would cut a central server query load of 50,000 requests per hour to about 4,000 per hour. And if the 7,500 queries per hour request is a more accurate estimate, by increasing the cache time, you'd decrease the central server workload from 7,500 to about 600 queries per hour.

If you increase the cache time, check the local cache first and then query the local log file second BEFORE going to the central server, you are effectively spreading that 7,500 query per hour workload out across thousands of servers. In doing that I bet you'd eliminate 95% of the load on your central server and everyone who uses your product would see much better performance.

In my mind, that's definitely worth thinking about.

So tell me, what have I missed here, pedigree? Where has my reasoning gone wrong?

I hope this helps.