Add-On Releases - SpamBuster: Identify, log and remove spam posts on your forum

This is the v3.6 version of SpamBuster that I wrote for vBulletin v3.5.x This version is mainly bug fixes.

What does it do?
It spots posts that could potentially be spam then (depending on option settings) creates a thread alerting site admin/moderators about the post and/or hides the post from view.

How does it identify the spam?
All off the posts for users with a post count of less than 10 (configurable) have their posts checked against a number of rules. Each rule has a different "score". If the total score for the post exceeds a specified limit then the post is deemed to be spam.

Can I add my own rules?
The installer file has a default rule set with it, you may wish to customise the rules in the options section of the AdminCP. The rules system is fairly simple to add to, in simple terms each rule is a perl regular expression and a score.

How do I install it?
Import the product via the Product Manager in the AdminCP. The options group and plugins will be added to your system. Now would be a good time to click the install link on here to record the fact that you've installed it.

How do I configure it?
Head to the "SpamBuster" option group in the "vBulletin Options" section of your AdminCP. Fill in the relevant fields. Further documentation will follow but it should be obvious what each setting does for the moment.

How do I upgrade?
Just run a regular install. It's worth backing up your settings particluarly the rules before performing the upgrade. If you have any problems I recommend removing the product then installing it from scratch.

How can I test it?
Try posting a new thread/post with some trigger values in it. The string cialis should do the trick.

Help it's broken my forum!
Erm, that's not an intended feature. Disable all six of the plugins that are part of the product "SpamBuster" and post an error report to this thread. The forum will work fine.

Why doesn't it do x,y or z?
Because I've either not thought about it yet or I've not written it yet. Please drop a post into this thread with any (sensible!) suggestions.

Can I use it commercially?
You're free to use it for whatever purpose comercially/privately as long as you don't resell the code. It's released under the LGPL, see the link below for futher info. If you find this plugin useful and you'd like to make a donation please use the "donate" button below.


click to donate


This software is licenced under the CC-GNU LGPL.


Changelog

• 8th November 2007 - v1.1.2 - New thread redirect bug fixed
• 14th August 2007 - v1.1.1 - Multiple post bug fix
• 13th August 2007 - v1.1.0 - Important bug fixes, updated rules file
• 15th June 2006 - v1.0.1 - Updated list of rules uploaded as a text file
• 6th April 2006 - v1.0.0 - Stable release: code remains the same but the rules file has changed.

[cdoyle]

I had this on my 3.5 forum, and we are about to upgrade to 3.7.
from what I'm reading this won't work with 3.7?

That really stinks, this is such a great hack, it worked wonderful on our forum.
I'm going to really miss it.

[PossumX]

I am running 3.7.1 PL2 and it is still working fine. Upgraded from 3.6.x all the way through, and it's still working.

3.7.2 PL1 upgrade is next, so we'll see what happens.

[Mecho]

Originally Posted by PossumX

I am running 3.7.1 PL2 and it is still working fine. Upgraded from 3.6.x all the way through, and it's still working.

3.7.2 PL1 upgrade is next, so we'll see what happens.

go for it , it works fine in 3.7.2 patch1 .

[septimus]

EXCELLENT Mod, using it on two forums and am very happy with it.

One issue: It seem to sporadically fail to post up the report post. I can make it put up the report post with a test spam thread or post, but occasionally it just up and doesn't. I'm not sure exactly why, but I suspect that when there's a post that contains a massive number of keywords it might be choking on it?

For now we're just keeping an eye on the moderation queue, but it really would be more convenient to be able to have the report thread posting work reliably.

Any ideas how I can go about troubleshooting this?

[gr8dude]

Hi everyone, I've been using SpamBuster for a while and it made my life significantly easier. I thought of an improvement, I implemented it on my forum but the results are not as expected.

My idea was simple - add a regular expression that detects any string that has at least 3 URLs in it.

The regexp I'm using is

regexp:a_not_too_many_links:15:any:"/(http:\/\/.*){3,}/i"

If I test it with online regex tools, any "bad" string that should be busted is busted; however, when the same regexp is used by SpamBuster - messages are still getting past the filter.

I don't understand what is causing this. My guess is that it has something to do with the priority of the rules used by SpamBuster?

Unfortunately I am not a skilled PHP programmer, but I think I would be able to figure it out if I could step through the code with a debugger (as one can do with other languages) and see the state of each variable as new instructions are executed.

Any thoughts?

[UK Jimbo]

Hi - it's a long time since I've worked on this plugin. For general spam filtering in recent version of vBulletin I'd suggest using the built in Akismet powered spam filtering.

The only way I can think of checking for three + URLs with a regexp is to use something similar to:

(untested)

/.*http:\/\/.*http:\/\/.*http:\/\/.*/

James


edit:

Perhaps try:

/(.*?http:\/\/.*?){3,}/i

for your regexp too

[gr8dude]

Thanks for the swift response.

This is what I use as a testbed:

• http://www.spaweditor.com/scripts/regex/index.php
• The following test string "this is http://link a test http://link http://link"

Every regexp you mentioned, as well as the one I devised myself works fine; i.e. the regexp tester reports that the given string matches the regexp.

Now, neither of them is effective when used with SpamBuster. I made this regexp the first one in SpamBuster's list - so I am certain this is not related to the order in which the rules are written in the config.

Maybe it can be understood by adding some debug statements to the plugin's code and watching the flow of its execution?


Thanks for suggesting Akismet, but my optimism faded away in just a few seconds - I saw that it is marked as "installed", therefore it means that some time ago I tested it and was not satisfied with the results and that my conclusion was that "SpamBuster is better" (-:

[Simon Lloyd]

I'd like to use this update version on vb 3.5.4 would it work?

[gr8dude]

UK Jimbo, I understood the cause of the problem.

The used regexp contains the ':' character (inside "http://").
When spambuster parses the rules, it uses ':' as a separator to explode the string into bits, ex:

regexp:a_nike_dunk:5:any:"/nike dunk/i"

Basically, the culprit is in this line of the sb_parse_config function:

// split up the line
$bits = preg_split('/(?<!\\\\):/',$rule);

This explains why the regexp is not found in the string - the regexp is simply not the one we thought we were using.

Suggested fix:
Use '::' as a separator in each rule, rather than ':'.

Replace

// split up the line
$bits = preg_split('/(?<!\\\\):/',$rule);

With

// split up the line
$bits = preg_split('/(?<!\\\\)::/',$rule);

Note: regular expressions are quite difficult to wrap one's brain around (-: Especially if I haven't written it myself, or if the regexp was written a long time ago. I didn't try to parse this regexp myself, I just guessed what it does and added another ':' and prayed for the best (-:

I don't know what the rationale behind that choice was, but isn't it easier to use this instead? It is straightforward and requires no belief in external forces :-)

$bits = explode('::', $rule);

My local tests are successful, below I have included the source of the file I am using for local tests. It is a stripped down version of Spambuster, the rules and the tested post are taken straight from this files, so you can make quick tests without involving VBulletin itself.


Can you share your thoughts on this, and perhaps update Spambuster to include this fix?

<?php


function sb_parse_config() {

	
	$rules_raw = <<<TEST
#Anything below this is added by Alex Railean
#regexp:a_jewels:5:any:"/^jewel/i"
#regexp:a_runescape:5:any:"/runescape/i"
#power leveling with spaces between words
#regexp:a_wow_powerleveling:10:any:"/power\s*leveling/i"
#regexp:a_wow_gold:5:any:"/gold/i"
#regexp:a_nike_dunk:5:any:"/nike dunk/i"
#regexp:a_hot:5:any:"/hot/i"
#regexp:a_manga:5:any:"/manga/i"
#regexp:a_jordan_shoes:5:any:"/jordan/i"
regexp::a_url::5::any::"/.*http:\/\//i"
regexp::a_gay::6::any::"/gay/i"
regexp::a_not_too_many_links::15::any::"/(http:\/\/.*){3,}/i"	
TEST;
	
	
	$rules = preg_split('/\r?\n/', $rules_raw);
	//echo var_export($rules, TRUE);
	
	$log = fopen("spam.log", "a");
    fwrite($log, strftime ("%c", time()) . "\n");
	$tmp =  var_export($rules, TRUE);
	fwrite($log, $tmp);
	
	$data = array();

	foreach($rules as $rule) {

		// comment lines
		if( strpos($rule,'#') === 0 )
			continue;

		// split up the line
		$bits = preg_split('/(?<!\\\\)::/',$rule);
		$bits2 = explode('::', $rule);
		fwrite($log, var_export($bits, TRUE));
		fwrite($log, "----\n");
		fwrite($log, var_export($bits2, TRUE));
		fwrite($log, $rule);

		// need the right number of arguements
		if( count($bits) < 5 )
			continue;

		for($i=0;$i<count($bits);$i++) {
			if( preg_match('/^"(.*)"$/', $bits[$i], $m) )
				$bits[$i] = $m[1];
		}


		$test=array();
		$test['type'] = array_shift($bits);
		$test['name'] = array_shift($bits);
		$test['score'] = array_shift($bits);
		$test['field'] = array_shift($bits);
		$test['data'] = $bits;

		$data[ $test['name'] ] = $test;
	}
	
	 $tmp = var_export( $data, TRUE );
	 echo $tmp."<br><br><br>";

	return $data;
}

//echo sb_parse_config();


// used to perform the test on the post
// function sb_test(&$obj,$table=null) {
function sb_test() {

	$hits=array();

	// no need to worry about most posts
	// if( $GLOBALS['vbulletin']->userinfo['posts'] > $GLOBALS['vbulletin']->options['spambusterpostcount'] )
		// return false;

	// parts of the post
	$req = array();
	// $req['title'] = $obj->fetch_field('title',$table);
	// $req['body'] = $obj->fetch_field('pagetext',$table);
	// $req['any'] = $req['title'] ."\n". $req['body'];
	$req['title'] = "title";
	$req['body'] = "nike tn (http://www.nikemaxtn.com/)    chaussure nike  (http://www.nikemaxtn.com/)   nike (http://www.nikemaxtn.com/)";
	$req['any'] = $req['title'] ."\n". $req['body'];

	// fetch the list of tests
	$tests = sb_parse_config();
	// run each test
	foreach($tests as $test) {
		//echo "#TEST  ".var_export($test). "<br>";
		echo "TEXT ".$req[ $test['field'] ]."<br>";

		$test_pass=false;

		// regular expression test
		if( $test['type'] == 'regexp' ) {
			echo "RGXP ". $test['data'][0]."<br>";
			$test_pass = @preg_match($test['data'][0],$req[ $test['field'] ]);
			echo "RSLT [".$test_pass."]<br><br>"; 
			//echo $req[ $test['field'] ] ."  ". $test_pass. "<br>";
		}

		// record the test if it was a hit
		if( $test_pass ) {
			$hits[ $test['name'] ] = $test['score'];
			$hits['total'] += $test['score'];
		}
	}

	return $hits;
}

sb_test();

?>

It looks messy, I apologize for that; PHP is simply not my cup of tea...

[gr8dude]

Seeing that there is no activity on this front, I attached my version of SpamBuster, with the change applied to it.

Note that the rules were changed, and that if you install it above the existing version you will have to manually update the rules via the administration panel.

I've been testing it since the time of the last message in this thread and it works as expected, no side effects were revealed.