vbSpamBuster v0.2

[UK Jimbo]

Following the release of vB SpamBuster v0.1 (https://vborg.vbsupport.ru/showthread.php?t=76513) earlier in the week I've added some of the improvements based on the discussion about it.

This version is a complete rewrite over v0.1 and adds the major features:

Moves spam into the moderation queue rather than giving the user a "permission denied error"
Has a highly configurable scoring system


As you can see the ruleset is fairly small at the moment I still think that it will catch a lot of spam posts. Any help/feedback from people running the system and either getting "false positives" or want to add any better rules to the default list please let me know.

Installation instructions

Download spambuster-libs.php and spambuster-rules.php.

You may wish to edit the define values at the top of the spambuster-libs.php file. Later on you might want to edit the rules in spambuster-rules.php too.

Upload spambuster-libs.php and spambuster-rules.php into your includes directory.

In both newthread.php and newreply.php find:
require_once('./includes/functions_bigthree.php');
after it add:
require_once('./includes/spambuster-libs.php');

In newthread.php find:
verify_forum_password($foruminfo['forumid'], $foruminfo['password']);
after it add:
// do a spambuster test
sb_test($_POST['subject'],$_POST['message'].$_POST['WYSIWYG_HTML']);

Now in newreply.php find:
$threadid = intval($_REQUEST['threadid']);
after it add:
// do a spambuster test
sb_test($_POST['title'],$_POST['message'].$_POST['WYSIWYG_HTML']);

As ever any feedback would be gratefully received.

[Polo]

Thanks for sharing :) UK jimbo

[oldengine]

Man I'd love to add a whole list of expressions to this kind of code!

==============================
visit our website

visit our web site

promotional

marketing

campaign

thousands

New web site

Urgent! Please Read!

work at home

earn a living

earn quickly

join our company

Fire Your boss

[Azhrialilu]

Just a couple then, huh? LOL

Great idea *clicks install*

[oldengine]

I should explain that I come to vBulletin after running eight years of WebBBS on my server. That script has a "naughty words" function that allows you to have a file of words or phrases that then kicks those posts into the moderated bin as they come in.

This has worked excellently for Nigerian scam posts and the typical overnight spam crap that gets posted from the far east. In fact, it worked so well that I created a duplicate function for the blind email system. This stopped Nigerian scam responses to my classified ads cold!

This is the first time that a hack to vBulletin has even come close to solving these problems. UK Jimbo, I support your work on this hack and I have subscribed to this thread. I realize that this was initially set up to block URL links, but if you have an example as to how my phrase word file could be coded into $spam_exprs, I'd surely appreciate it. Otherwise the number of $spam_exprs in the code could get lengthy.

EDIT: I have tried this hack in my testvb and it works fine. Now for the refinements of kicking those posts into the moderated mode.

[nso]

It would be really great if you could implement the definition of the spam_exprs into the admin control panel. Also a logfunction would be nice to have.

There is very little spam on my board, but I still very much support any effort in this direction. Great work

[UK Jimbo]

Thanks for the feedback everyone.

As time permits I'm going to be looking at:

adding the detection to newreply.php
dropping bad posts into a moderation queue
logging/reporting trapped posts
configuration via the admincp (with possible word list import/export)

[nso]

adding the detection to newreply.php
[/list]
Actually, to me it seems like adding your newthread.php code in the corresponding place in newreply.php would do the trick

[fly]

With all due respect, wouldn't it just be easier to not let new users post URLs?

[UK Jimbo]

With all due respect, wouldn't it just be easier to not let new users post URLs?

Yup - but you can't do that with vBulletin out of the box can you?

Just drop the regular expression http:// into the config for this hack and hey presto you'd be able to do that.

[oldengine]

As time permits I'm going to be looking at:

configuration via the admincp (with possible word list import/export)


Make sure that the list isn't just words. As you can see by my list of spam lines above, the rejection power is made by the combination of words into spam phrases.

My actual list is hilarious and lengthy, all taken from actual posts and emails.

[UK Jimbo]

Make sure that the list isn't just words. As you can see by my list of spam lines above, the rejection power is made by the combination of words into spam phrases.

My actual list is hilarious and lengthy, all taken from actual posts and emails.

For "words" read "regular expressions". Is in the example code above it'll be using perl regular expressions which are very powerful when it comes to pattern matching.

[Snow]

Yup - but you can't do that with vBulletin out of the box can you?

Just drop the regular expression http:// into the config for this hack and hey presto you'd be able to do that.
Does this hack work on thread titles as well or is it just for spam in the post itself? The reason I ask is that I had installed (a while back) a different hack that prevented new users from posting urls in posts and signatures but as a result, spammers started posting the urls in the thread titles and their profiles. It was very common to see "visit my homepage from my profile to find out more".

[UK Jimbo]

Does this hack work on thread titles as well or is it just for spam in the post itself? The reason I ask is that I had installed (a while back) a different hack that prevented new users from posting urls in posts and signatures but as a result, spammers started posting the urls in the thread titles and their profiles. It was very common to see "visit my homepage from my profile to find out more".

Good point.

It'll be easy to include post/thread titles in the list of fields checked so I'll add that in. Good point about profiles. I'll look to see if this is possible but might make things a bit more tricky.

[Mink_]

To get that working in the ACP, I suggest you just make a new database table and call the values into an array with mysql_fetch_array().

Also, I suggest you use the similar_text() function and allow the user to configure what percent 2 words have to be for the replacement to happen. That can get rid of peoblems with people figuring out how to get around the hack by saying things like "Vist my wbsite."

[UK Jimbo]

I've made some progress with this today. It's now dropping threads into the moderation queue.

In the next few days I'll put another release up on here with all the changes I've made and will be reading the regular expressions out of a separate text file. After that's sorted the next release will concentrate on admincp integration.

I suggest you use the similar_text() function and allow the user to configure what percent 2 words have to be for the replacement to happen.

I've never spotted that function before! I think I may support both regular expression and similar_text phrase lists. Watch this space.

[UK Jimbo]

Following some further thought and after having used spamassassin (http://www.spamassassin.org/) a fair bit in the past I think that I'm going to run with a similar rule based system.

A config file has a list of rules within it. Each rule has a score which is added to an overall score for the post should that rule match the post.

At the end of testing a post for "spaminess" the overall score of the post is compared to a configurable "threshold" score. If the score for a post is over the threshold then it's deemed to be spam.

This setup, coupled with perl regular expression and similar_text support should make for a fairly simple but highly configurable system.

After there's a stable release of this code I'd hope that a number of us could worth together to produce a decent standard list of patterns that people can use out of the box. I'm going to need some help with that stage as the forums I admin have fairly low amounts of spam compared to some other sites.

As an aside SpamBuster nipped 8 spam posts in the butt this morning on visordown.com (http://www.visordown.com/forums/) :)

[UK Jimbo]

Following the release of vB SpamBuster v0.1 (https://vborg.vbsupport.ru/showthread.php?t=76513) earlier in the week I've added some of the improvements based on the discussion about it.

This version is a complete rewrite over v0.1 and adds the major features:

Moves spam into the moderation queue rather than giving the user a "permission denied error"
Has a highly configurable scoring system


As you can see the ruleset is fairly small at the moment I still think that it will catch a lot of spam posts. Any help/feedback from people running the system and either getting "false positives" or want to add any better rules to the default list please let me know.

Installation instructions

Download spambuster-libs.php and spambuster-rules.php.

You may wish to edit the define values at the top of the spambuster-libs.php file. Later on you might want to edit the rules in spambuster-rules.php too.

Upload spambuster-libs.php and spambuster-rules.php into your includes directory.

In both newthread.php and newreply.php find:
require_once('./includes/functions_bigthree.php');
after it add:
require_once('./includes/spambuster-libs.php');

In newthread.php find:
verify_forum_password($foruminfo['forumid'], $foruminfo['password']);
after it add:
// do a spambuster test
sb_test($_POST['subject'],$_POST['message'].$_POST['WYSIWYG_HTML']);

Now in newreply.php find:
$threadid = intval($_REQUEST['threadid']);
after it add:
// do a spambuster test
sb_test($_POST['title'],$_POST['message'].$_POST['WYSIWYG_HTML']);

As ever any feedback would be gratefully received.

[UK Jimbo]

v0.02 released at https://vborg.vbsupport.ru/showthread.php?t=76756

[Polo]

I dont think you can Make a new thread when having a new version of a hack... I think you should use the same thread... i'm not sure, but I think I read that somewhere... anyways, let me go look around...

[UK Jimbo]

<a href="https://vborg.vbsupport.ru/announcement.php?f=38&announcementid=18" target="_blank">https://vborg.vbsupport.ru/annou...ouncementid=18</a> is from the main release boards (and doesn't mention versions).

I couldn't find anything for the Beta forums either.

[Erwin]

https://vborg.vbsupport.ru/announcement.php?f=38&announcementid=18 is from the main release boards (and doesn't mention versions).

I couldn't find anything for the Beta forums either.
I added it to the rules now. But most hackers know that this is frowned upon. Why create a new thread for a new version? Some hacks have 20 versions - do they get 20 hacks? :)

[UK Jimbo]

I added it to the rules now. But most hackers know that this is frowned upon. Why create a new thread for a new version? Some hacks have 20 versions - do they get 20 hacks? :)

Fair play - I'd not used the forums enough to realise this was frowned upon.

Please feel free to merge the two threads in this forum.

[oldengine]

Hopefully, the new thread and subsequent re-merging didn't throw cold water on this hack. It is (to me) one of the more significant hacks on this site. Unfortunately, it is quite hard to find once removed from the list of "Latest Hacks" on the home page.

[UK Jimbo]

Hi oldengine - thanks again for the support.

Michael Morris and I will be working together to merge SpamKiller and SpamBuster together. We've had one brainstorming session over MSN Messenger but will be having more in the coming days and hopefully releasing something soon.

https://vborg.vbsupport.ru/showthread.php?t=76800

as an aside no spam has been sucessfully posted on visordown.com in the last four days :D

[Odoin]

Hi Guys...

Would this work with the NNTP mod? I get tons of spam from the NNTP groups.

Thank you..

[oldengine]

UK Jimbo: I fear that the inclusion of an external site download of material has overburdened this idea.

What started out initially would work within my server and be updated by myself only as needed. The spam quantity that I get does not justify an automated update. You saw part of my word list in an earlier post. That same word file could include the heart of a domain name to block a URL as well.

Is the hack at the begining of this thread complete and sufficient for a self-maintained anti spam system based on a word/phrase list kept in a file?

[White Knight]

Just to give you a heads up. I installed it and its working great. Thanks for the Hack!

[oldengine]

Clicks install, finally. (Just had someone post some crap that I'd rather not have see the light of day.)

I guess I've figured it all out.

EDIT:

# contins the word free anywhere
regexp:a_free:5:any:"/free/i"

So, to add "work at home" as a phrase:

regexp:a_free:5:any:"/work at home/i"

Correct? (of course we could make that one a 10.) :D

[007]

It would be cool if it would autoban users when a certain number of their posts in a row are flagged as spam. :)

[oldengine]

I'm starting to panic! This and a lot of other good mods are either going to go down the drain or I'm going to stay on vB 3.0.9 and forget about 3.5.

Now I see why there are still users on vB 2. :disappointed:

[UK Jimbo]

As soon there is a gold release of 3.5 and we've migrated the forum I use to it then I'll port this mod over to 3.5 (to save my sanity patching in this and a load of other mods every time the forum software gets upgraded).

:)

[oldengine]

Glad to hear this. Spambuster is a valued asset to vBulletin!

[motorhaven]

There's a major hole in this hack...

Create a new thread or post without a spam link or keyword. Post appears (as it should). Now edit the post and put in the spam link/keyword. Post is live, instead of in the moderation queue.

[motorhaven]

Okay, here's the fix:

In editpost.php look for:
require_once('./includes/functions_editor.php');

Below that add:
require_once('./includes/spambuster-libs.php');

In editpost.php look for:
build_post_index($postid , $foruminfo);

$url = "showthread.php?$session[sessionurl]p=$postid#post$postid";
eval(print_standard_redirect('redirect_editthanks' ));


Above that add:
sb_test( $edit['title'], $edit['message'] );
if ( $foruminfo['moderatenewpost'] == 1 )
{
$thepost = $DB_site->query_first("SELECT * FROM " . TABLE_PREFIX . "post WHERE postid = $postid");

// First post in thread?
if ( $thepost['parentid'] == 0 )
{
$hidetype = "thread";
$DB_site->query("UPDATE " . TABLE_PREFIX . "thread SET visible=0 WHERE threadid = $threadinfo[threadid]");
}
else
{
$hidetype = "post";
}
$DB_site->query("UPDATE " . TABLE_PREFIX . "post SET visible=0 WHERE postid = $postid");

$DB_site->query("DELETE FROM " . TABLE_PREFIX . "moderation WHERE postid = $postid");

$DB_site->query("INSERT INTO " . TABLE_PREFIX . "moderation (threadid, postid, type )
VALUES ($threadinfo[threadid], $postid, '$hidetype')" );
}


Could be done more efficiently but I threw it together in 15 minutes and it works.

[UK Jimbo]

Thanks :)

I'll make sure that it's not an issue when I upgrade the code to work with vBulletin 3.5.

[UK Jimbo]

I've not forgotten about v3.5ing this (and possible enhancing it a bit).

Watch this space...

[oldengine]

Definitely been watching. It's my gateway to 3.5. :)

[oldengine]

<bump>

[nwelsh]

you know what, this hack is really good for the contact us form!

[-=Sniper=-]

UK Jimbo; would love to see a 3.5.2 version :)

[UK Jimbo]

Sorry a v3.5.x version hasn't materialised yet.

I'm on an extended overseas trip at the moment and it's hard to find the time for it. It's definitely on my TODO list :)

[Ski-Whiz]

Can this sort of thing also be added to the PM system? We are getting people who are trying to be sneaky, and pm members through the pm system as well.

Unless members come forward and tell you, the site won't know about it.

So adding a checker in the pm system would be great!

Thanks!

[motorhaven]

Here you go....

In spambuster-libs.php, look for this at the bottom of the file:

?>

Above it, add:


// used to perform the test on emails/pms
function sb_test2($title,$body,$recip="")
{
$hits=array();

// no need to worry about most posts
if( $GLOBALS['bbuserinfo']['posts'] > SB_MIN_POSTS )
return false;

// parts of the post
$req['title'] = $title;
$req['body'] = $body;
$req['any'] = $title ."\n". $body;

// fetch the list of tests
$tests = sb_parse_config(SB_CONFIG);

// run each test
foreach($tests as $test) {

$test_pass=false;
// regular expression test
if( $test['type'] == 'regexp' ) {
$test_pass=preg_match($test['data'][0],$req[ $test['field'] ]);
}

// record the test if it was a hit
if( $test_pass ) {
$hits[ $test['name'] ] = $test['score'];
$hits['total'] += $test['score'];
}
}

// it isn't spam
if( $hits['total'] < SB_TRIGGER )
return false;

$ret=array();
$ret['hits'] = $hits;

// build a pretty string of the scores
$pairs=array();
foreach($hits as $k => $v) {
if( $k == 'total' )
continue;

$pairs[] = "$k=$v";
}
$ret['HITS_STR'] = implode(', ',$pairs);

$ret['USER'] = $GLOBALS['bbuserinfo']['username'];
$ret['MESSAGE_TITLE'] = $title;
$ret['MESSAGE_BODY'] = $body;
$ret['RECIP'] = $recip;

// send alert emails
sb_send_mail2($ret);

return true;
}


function sb_send_mail2($vars) {

$mail=array();
$mail[] = "This is an automated email from vB SpamBuster";
$mail[] = "";
$mail[] = "The user <%USER%> has just tried to email or PM the following message to <%RECIP%>:";
$mail[] = "";
$mail[] = "***********************************************";
$mail[] = "";
$mail[] = "The vB SpamBuster system deemed it to be spam after it passed the following tests:";
$mail[] = "<%HITS_STR%>";
$mail[] = "";

$msg = implode("\n",$mail);

foreach($vars as $k => $v) {
$msg = str_replace("<%$k%>",$v,$msg);
}

$emails = explode(' ',SB_ALERT_EMAILS);
foreach($emails as $email) {
vbmail($email,'vB SpamBuster PM or Email Alert',$msg);
}
}


(Note, I don't include the message title/body in the alert due to privacy issues.

In private.php, look for:


$title = addslashes(htmlspecialchars_uni(fetch_censored_tex t($pm['title'])));
$message = addslashes(fetch_censored_text($pm['message']));
$signature = intval($pm['signature']);
$iconid = intval($pm['iconid']);
$disablesmilies = iif($pm['disablesmilies'], 0, 1);


Below that add:


// 11-26-2005: see if message passes spam trap filters
if ( sb_test2( $title, $message, implode(' ', array_keys($sendto) ) ) )
{
// fails spam test
}
else
{


Next, look for


}

$url = "private.php?$session[sessionurl]";
eval(print_standard_redirect('pm_messagesent'));


Change it to


}
}

$url = "private.php?$session[sessionurl]";
eval(print_standard_redirect('pm_messagesent'));



In sendmessage.php look for (towards the bottom of the file):


eval(fetch_email_phrases('usermessage', $destuserinfo['languageid']));


Below that add:

// Test against spam trap filters
if ( sb_test2( $emailsubject,$message, $destuserinfo['email'] ) )
{
// Message fails filters...
// parse this next line with eval:
$sendtoname = $destuserinfo['username'];
eval(print_standard_redirect('redirect_sentemail') );
return;
}

[oldengine]

This is a great idea, but needs one modification.

Spam traps are not a "one shoe fits all" word list. The type of spam we tolerate posted on the board may be different than the spam words tolerated in emails. Therefore, a different spambuster-rules.php will be needed for the email spam.

There are items placed in my spambuster-rules.php that I would rather users communicate by email to eachother instead of in the open for all board readers to see. Copying spambuster-rules.php to spambuster-mail-rules.php and altering the contents would get the job done.

Hey! I'm waiting to upgrade to 3.5 because of SpamBuster!
EDIT: No longer. 3.5.3 is installed.

[UK Jimbo]

I've finally found the time to sort this out. 90% of the code is ready now and there'll be a beta release that just uses plugins and admincp configuration in the next few days.

[UK Jimbo]

Beta v3.5.x version here (https://vborg.vbsupport.ru/showthread.php?t=109258)