Akismet - spam detection

[El_Muerte]

Akismet (http://akismet.com) is an spam detection engine that is mostly used by blogs to check the comments for spam. But it can also be used for forum posts. It is not a definite protection against spam.

This plugin for vBulletin 3.6 will check every thread and reply if it is considered as spam. If it is spam it will put the offending thread or reply in the moderator queue.

In order to use Akismet you need to get a API key. You can get one by signing up on WordPress (http://wordpress.com). The Akismet service is free for personal use. Check the Akismet website for more information.

This vBulletin plugin is available under the terms of the GNU Lesser Public License.

Installation instructions are available in the ZIP archive.

Features

Scan for spam in new threads and post
Scan content edits
Automatically put identified spam in the moderation queue
Report posts as spam or ham (individual post or through inline moderation)
Limit spam detection to post count and user groups
Extendable through hooks


Changelog
v1.3

Added an akismet activity log in the admincp


v1.2

Fixed an invalid URL issue when using cURL


v1.1

Option to send parsed post data
Option to send additional webserver request data for the initial spam check
Removed product version query
Added extra information when reporting a post
Only check API key when entered in the vBulletin options, not for every Akismet request


v1.0

Various bug fixes: report post submitted wrong data; few missing phrases
More code documentation
This plugin now exposes some hooks that others can use to extend the feature set. Like give users infractions when posts are considered spam.


v0.3

Added options to exclude posts from checking based on user's postcount and usergroups
Added mass reporting of ham/spam through the inline moderation system.
Improvements of the internal code


v0.2

Check spam status on edits
Add report as ham/spam functionality (requires template edit)

[El_Muerte]

I welcome any feature suggestions and code additions. Feel free to help me improve this mod.

[Zia]

nice to see...hope we'll find it very much usefull

[BadgerDog]

Installed..... :)

Thanks for your efforts.... :up:

I'm so sick of banning and deleting spammers who register and login, starting threads about prescription drug sales or pornographic sites.

I hope this works as I've never handled moderated threads before. I'm not even sure how that function works in vBulletin.

Regards,
Badger

Edit: Question: NM, got it figured out and now have "Quick Moderation" page appearing on Home Page. Thanks.

[El_Muerte]

Ok, version 0.2 has ham and spam reporting.

When you encounter a spam message that was not detected simply press the red flag to submit that message as spam. This will increase the chance that the message will be detected as spam the next time. In the same way you should report a message as ham (green flag) when it was detected as spam. A successful report will automatically mark the post and put it in or remove it from the moderation queue. Of course only forum moderators will be able to report posts as ham or spam.

The red exclamation icon tells you the message has been identified as spam.

[BadgerDog]

In the process of updating to 0.2.... :)

It says we need to modify the template postbit.....

Should we not also modify the template postbit_legacy as well, if boards use it?

Thanks

Regards,
Badger

[BadgerDog]

Of course only forum moderators will be able to report posts as ham or spam.

Since there is no place in this mod that I can find to identify who is a forum moderator or not, can I assume you've defaulted to UserGroup numbers 5,6,7 (Senior Moderator, Admin, Moderator) as the basis for this?

Thanks :)

Regards,
Badger

ps: Can I make a suggestion? Instead of the phrase "ham", why not call it "Not Spam" to avoid confusion?

[El_Muerte]

It says we need to modify the template postbit.....

Should we not also modify the template postbit_legacy as well, if boards use it

Er.. yes, if your board uses it you should also update it.

Since there is no place in this mod that I can find to identify who is a forum moderator or not, can I assume you've defaulted to UserGroup numbers 5,6,7 (Senior Moderator, Admin, Moderator) as the basis for this?

no usergroup check, it simply relies on vBulletin's API. if the user has privileges to moderate posts in a forum.

ps: Can I make a suggestion? Instead of the phrase "ham", why not call it "Not Spam" to avoid confusion?

"ham" is the going name for "not spam", you could simply change the phrases that use "ham".

[BadgerDog]

Er.. yes, if your board uses it you should also update it.

Done... :)

no usergroup check, it simply relies on vBulletin's API. if the user has privileges to moderate posts in a forum.

Yup, tested it various ways logging in as just a normal user and it works fine... thanks.. :)

"ham" is the going name for "not spam", you could simply change the phrases that use "ham".

I didn't know that... must be getting old... ;) I think 'll change the phrases locally... :up:

Nice mod... now I'm just waiting for a spammer to show up so I can see what happens. :D

I assume this doesn't put a performance (or bandwidth consumption) drain on the system when it's checking externally against their database?

Thanks again...

Regards,
Badger

[sensimilla]

I used to have it installed, its a great modification, thanks for the update.

[El_Muerte]

I assume this doesn't put a performance (or bandwidth consumption) drain on the system when it's checking externally against their database?

Depends on what you call a drain. Post will only be checked when they are created or edited. Only the post body (and a few other pieces of information) will be send to the akismet servers. This traffic is much less than an average user viewing the forum.

Starting from 0.3 you can limit the spam checking a bit. E.g. don't check spam for users in certain usergroups and/or check users with less that X posts (defaults to 50). Spammers are usually new users. No real need to check with established users.
50 might be a bit on the high side since it would take the user 50*post delay before he will get around the spam checking. For example if your post delay is 30 seconds it will take the spammer at least 25 minutes before he can start spamming the forum and get away with it. Of course the each post being made must be a valid post.

[AlexanderT]

Nice mod, especially like your improvements!

[BadgerDog]

Depends on what you call a drain. Post will only be checked when they are created or edited. Only the post body (and a few other pieces of information) will be send to the akismet servers. This traffic is much less than an average user viewing the forum.

Starting from 0.3 you can limit the spam checking a bit. E.g. don't check spam for users in certain usergroups and/or check users with less that X posts (defaults to 50). Spammers are usually new users. No real need to check with established users.
50 might be a bit on the high side since it would take the user 50*post delay before he will get around the spam checking. For example if your post delay is 30 seconds it will take the spammer at least 25 minutes before he can start spamming the forum and get away with it. Of course the each post being made must be a valid post.


Great feedback, that's what I suspected about bandwidth and load.... :)

Your 0.3 enhancements (and logic) are right on the money. Spammers on our systems are usually one-timers. They show up, go through the registration process including even doing the email validation, then they post 1 or 2 messages selling porno, drugs or cameras in a couple of public forums. We catch them, ban them and move on.

So, your usergroup check and amount of posts made measurements are exactly how to maximize the mod and minimize the activity.

Thanks for your efforts... :up:

Regards,
Badger

[BadgerDog]

Ok, in .3, I added the additional template change to showthread for in-line moderation, but I can't see anywhere what it does?

Edit: Never mind.... found it... was looking in the wrong place ... very nice design...

Thanks... :)

Regards,
Badger

[projectego]

/me clicks install

[El_Muerte]

Ok, this product is now complete. Everything I planned to add has been added. Other programmers can extend the functionality through the exposed hooks in the code.

[AlexanderT]

Some remarks:


post content is sent in bbcode format to Akismet; wouldn't it be better to parse it first?
verifykey is called each time a comment is posted; wouldn't it be sufficient to use verifykey at the time when the key is entered in admincp?
an additional query is used every time a comment is posted to determine the installed product version; wouldn't it be sufficient to use a define for the product version and to save the query?


Great add-on!

[El_Muerte]

1) I don't know. There is not much info on how akismet really check the content. And spammer often use standard messages for all spam runs, parsing to bbcode could result in different content for each forum.

2) Probably. The documentation isn't very clear on that subject. Then again, if the key is invalid it will probably be notified in the spam check or report.

3) I felt lazy, didn't want to update the define everytime I changed the file. Then again, it's a little effort for a more or less useless query.

[El_Muerte]

Corrected above points in version 1.1

[BadgerDog]

Just installed v1.1 and now I have an error under the AdminCP function....

API Key
The key required to interface with Akismet.
To acquire a key you have to create an account at WordPress

Error: You did not enter a valid value for this setting.

I have a valid key and haven't touched this field since I original installed the mod and received the key from Wordpress. The error appeared after installing v1.1.

Hope this helps find the bug....

Should I try to do an overwrite and replace v1.1 backwards with v1.0?

Regards,
Badger

[El_Muerte]

Really? I don't get that error for my key. Are you sure that key is correct? Nothing changed to the key validation since 1.0 (except that I removed revalidation during spam checking).

If you have the vBulletin debug functionality enabled you can see the response from akismet in the "debug message" dropdown list

ps, reverting to 1.0 should be possible

[BadgerDog]

Really? I don't get that error for my key. Are you sure that key is correct? Nothing changed to the key validation since 1.0 (except that I removed revalidation during spam checking).

Perhaps it doesn't like that change?

I applied for the key when you first released the program. Entered it into the field and haven't touched it since, all through your various upgrades. This error came up as soon as I installed v1.1

I went to the Wordpress web site and validated my key and it's entered correctly. I even cleared the field and started entering it one character at a time as their FAQ said that sometimes cut-n-paste causes problems. About 1/2 way through the error message came up.

Should I uninstall v1.1 and try to re-install, or simple re-install v1.0 over v1.0?

BTW, where do I find the DEBUG drop down list?

Regards,
Badger

[BadgerDog]

If you have the vBulletin debug functionality enabled you can see the response from akismet in the "debug message" dropdown list


Ok, I found how to activate DEBUG mode and at the bottom it says:

DEBUG MESSAGES (1) [akismet] Key = $RCSfile$ - $Revision: 17009 $
SQL Queries (10)

Is that what you're looking for?

Regards,
Badger

[BadgerDog]

ps, reverting to 1.0 should be possible

Ok, reverted back to v1.0 (importing over v1.1) and the error is still there... :(

After that, I completely UN-INSTALLED the mod and re-installed v1.1 and the error is still there. :(

Regards,
Badger

[El_Muerte]

be sure to also overwrite the functions_akismet.php when you upgrade/revert

anyway, there appears to be connections issues, the debug line should be either:
"Key = valid" or "Key = invalid"
if it doesn't contain any text then there was an error in connecting to the akismet server to validate the key.

the key isn't validated while you are typing, it will be validated when the text field loses focus, and when the options pages is loaded.

Besides, you can simply enter an invalid key and save the the options. The rest of the plugin will still try to check for spam and submit content despite a possible invalid key.

[BadgerDog]

be sure to also overwrite the functions_akismet.php when you upgrade/revert

Did that ... :)

anyway, there appears to be connections issues, the debug line should be either:
"Key = valid" or "Key = invalid"
if it doesn't contain any text then there was an error in connecting to the akismet server to validate the key.

the key isn't validated while you are typing, it will be validated when the text field loses focus, and when the options pages is loaded.

Besides, you can simply enter an invalid key and save the the options. The rest of the plugin will still try to check for spam and submit content despite a possible invalid key.

The message actually pops up during typing...

Perhaps if I sent you a PM and gave you access, you could debug the problem better?

Thanks

Regards,
Badger

[AlexanderT]

Corrected above points in version 1.1

Cool! :up:

[El_Muerte]

Version 1.1 has a bug when using cURL for communication (cURL will automatically be used when it's available). For some reason it worked perfectly on my system, but on an other system cURL didn't accept the broken url.
So if you have cURL installed on the system you should upgrade to 1.2, if unsure also upgrade.

Thank you BadgerDog for letting me use your board to figure out this problem.

[BadgerDog]

Thank you BadgerDog for letting me use your board to figure out this problem.

You're welcome .... :)

Installed v1.2....

All fixed and running smooth.... :up:

Regards,
Badger

[BadgerDog]

Ok, had my first spammer visit since installing this mod.

He posted two threads selling cameras and neither was "moderated" by the mod or "Akismet", so I submitted them to Akismet marking them as spam. This particular ad is quite common and they've visited and posted it before, so I guess the real test now is when they show up the next time, does the Akismet check recognize their next similar post as SPAM

Will advise ... :)

Regards,
Badger

[BadgerDog]

Update....

I had another spammer come back and post the same ad as I reported to Akismet using this mod, as I indicated in the above post.

Unfortunately, the exact same ad (even though reported) still was posted twice in two different forums and was not caught and moderated. :(

So, the conclusion is that something isn't working right, either with the mod itself, or with the interaction using Akismet for checking posts before they actually end up live in a forum.

I have again marked the posts as SPAM and the routine of this mod says it has reported them to Akismet, as well as the mod moved them to become moderated.

Not sure what I should do at this point for testing?

Regards,
Badger

[El_Muerte]

if you want to test if things are working create a user with the name: viagra-test-123
every post made by that users should be qualified as spam.

just to be sure I just check if everything is send correctly to the akismet server, and it is.
they are not very clear on how they detect spam and so far I haven't had a confirmed spam message (or even spam at all).
but if they use a learning system the detection should improve every time you mark content as spam.

but I'm sure the detection will become better as the user base increases, or at least I assume there are major differences between the content of blog spam and forum spam.

anyway, I'll get in contact with the akismet people and try to figure out some more stuff

[BadgerDog]

anyway, I'll get in contact with the akismet people and try to figure out some more stuff

Thanks for your support ... :)

I'd be interested hearing what they have to say, particularly why an ad with exactly the same content is reported using their format, yet when posted again it doesn't get recognized by their software routines.

That's a good idea about creating a test user called viagra-test-123... :up:

In fact, I have a private forum that I can use to create ads with various obvious spam phrases, so I'll try to create a bunch of new posts using that name as well as put content in them that should be picked up.

I'll let you know the results.... :)

Regards,
Badger

[BadgerDog]

Ok, I created that usename viagra-test-123 and tried a post.....

When I hit submit for the post, the screen cleared and there was a statement that went by so quick I could hardly read it. Something to the effect of my post was being submitted for moderation. They really should halt that "screen" with a button for OK to continue so people can see what it says. :D

Anyway, apparently it didn't like the post from username viagra-test-123 and did immediately put it into moderation que ... :up:

Here's the screen that I saw when I went to the que and checked it.

Posted By viagra-test-123
Forum - Library Post Preparation and Testing Area
Title - Message Testing spam........

Message

Testing new SPAM thingy...

Notes - Akismet reported this as spam

Action - Validate Delete Ignore

So, I guess the word viagara in the username was the catch phrase here, since nothing else in the subject or body of the post (except the word spam) was threatening enough to trigger a moderation event.

I'm going to do some more testing using a username that's not obvious and try to put some various phrases into the body of the posts that are obvious. ie: viagara.

I suspect that will work, but the real issue should be that once I've reported something as spam from my user site (web address), I would have thought that Akismet recognizes it going forward as spam, at least when associated with my site alone.

That's what I think it's NOT doing, because HUGE ads for cameras and cell phones being reported doesn't seem to be enough to get the same ads on their list.

Thanks again for your work. :)

Regards,
Badger

[BadgerDog]

Update again ..... :)

Ok, using the username Tester this time, I posted message with the phrase viagara in the BODY of the post and also created a second post with the phrase viagara in the SUBJECT header.

Neither one was picked up by Akismet.....

So, I guess it picks up viagara when part of the username, but nowhere else?

Hope this helps....

Regards,
Badger

Edit: Out of curiosity, I set the option "Send extra request information" to ON and tried the above tests in this post again, however, it made no difference and Akismet failed to spot it.

[El_Muerte]

The username 'viagra-test-123' is a testing feature, akismet will consider everything from viagra-test-123 as spam. It's just for testing purposes to check if the system is still properly communicating with the servers.

Anyway, just uploaded 1.3 which adds an activity log for akismet actions.

[BadgerDog]

V1.3 installed and functioning..... :up:

I can see new log function at bottom of "Statistics and Logs" area of AdminCp....

Will advise results....

Thanks... :)

Regards,
Badger

[BadgerDog]

Ok, serious issue..... :D

Just tried to post simple test message under username Tester and when I hit submit, got a white Vbulletin screen saying DATABASE ERROR with no other information other then the standard "Go Back" kind of thing.

When I back up, the message never got posted anywhere (even moderation) and I suspct it has something to do with trying to write to the database log?

Since I'm on a LIVE system, I've had to disable v1.3 version of mod in the AdminCP until you get a chance to look at it. I re-tested posting after I disabled the mod and the user Tester was able to post the exact same message with no error, so I assume by leaving the mod disabled the system will continue to operate without problems (although no Akismet checking).

Regards,
Badger

Edit: By the way, there were NO ENTRIES in the log at any time during this testing process, if that helps....

[El_Muerte]

oh crap, forgot to include the table prefix in the install query to create the table. get 1.3.1

[BadgerDog]

oh crap, forgot to include the table prefix in the install query to create the table. get 1.3.1

Installed v1.3.1 (Overwrite) .... :up:

Re-enabled mod...

Ran same simple post test with username tester as described above...

Got the exact same vBulletin database error.... nothing recorded in log...

Disabled mod and retested..... no error....

Hope this helps track down the databse error problem that began with adding the "logging" function...

Regards,
Badger

[El_Muerte]

wow.. I really ++++ed up this release.. just to be sure I've double checked version 1.3.2, it installs correctly on a fresh vbulletin install.

[BadgerDog]

Installed v1.3.2 and tested.... :)

No errors and I now have log entries.... :up:

I'll send via PM due to confidential data....

Thanks

Regards,
Badger

[El_Muerte]

great to hear it's starting to catch spam

[offline]

Awesome mod, I have used Aksimet spam detection on my 3.5vb and it has saved me from tons of spam. I am now up to date but I was wondering if I should tun on or off the send extra request information?

[yoyoyoyo]

Unfortunately I had to turn this off because many non-spam posts and threads were getting sent to the moderation queue. I looked to see what the posts had in common, or what would make them stand out as spam, and was unable to determine why they would be considered spam - no links posted, no strange keywords that migh seemingly be flagged for an odd reason, Would the extra request information help with this? Anything else I can do to make it less sensitive? I could turn off the "send posts/threads to moderation queue option, but then if I did that it seems it would make this hack kind of pointless.

[BobbyBig]

How about a Mod which exactly does the same but with Rules we can set ?
I am badly searching for that.

akisment doesnt seem to be effective enough for vbulletin.

[Hemanth]

How can I uninstall it from vB 3.6.5. It's not getting uninstalled and once enabled breaking the entire postings.. like duplicate posts, parse errors and even db errors.

Damn it.

[skippybosco]

Thank You. Working great on vB 3.6.8.

It seems that only the blog, user_ip and user_agent are required arguments for the API (note: not the actual message)

With that in mind, would it make sense to also use this to optionally check during the registration process as well?

I consistently see that the IP and/or email and name associated with flagged messages are associated with known spamming activities so there should be a fairly accurate hit ratio.

This would also save us the trouble of cleaning up the moderated posts and banning the users after the fact by either preventing them from signing up altogether or putting them in a n even more restrictive group.

[skippybosco]

A secondary suggestion would be utilizing this for checking Private Messages sent between users to prevent mis-use of bogus or compromised accounts.

[skippybosco]

For those looking for the registration checking functionality, I am investigating the use of the Check Proxy RBL on New User Registration (https://vborg.vbsupport.ru/showthread.php?t=131852) mod as a temporary solution to limit the registration of SPAM users before they post/PM.

Still interested in seeing this functionality rolled into the Akismet as well as checking of PMs :-)

[skippybosco]

Check Proxy was a no go on cutting down spam registrations..

Any suggestions from the community? Note that no actual posts are making their way to the board thanks to this great plugin, but I'd like to cut out the registration all together if possible and no combination of human detections have seemed to deter them.

[dutchbb]

Product import takes forever in 3.7 so probably not working. Please update this for 3.7, thank you.

[epolitica]

* Please update this for 3.7, thank you.

[Damien001]

will this work on vb 3.8?

Thanks

[BadgerDog]

Anyone tried this with vb4 after updating?

It really worked well for us under v3.68 and would like to keep using it...

Thanks .. :)

Regards,
Badger

[skippybosco]

Yes. Works well in VB4

[BadgerDog]

Thank you very much !!! :)

Can't tell you how effective this has been ...

Regards,
Badger

[BadgerDog]

Yes. Works well in VB4

Just noticed ... :)

Akismet option is built into vb4 4.03 ...

Did you use this add-on version, or the one built-in?

This add-on seems to have more options, including user group selections etc...

Thanks ..

Regards,
Badger

[gr8dude]

Is there a way to configure the plugin such that it automatically removes a message if it is flagged as spam?

My moderation queue is overloaded, it becomes impractical to monitor it...