Akismet - spam detection
 

Akismet is an spam detection engine that is mostly used by blogs to check the comments for spam. But it can also be used for forum posts. It is not a definite protection against spam.

This plugin for vBulletin 3.6 will check every thread and reply if it is considered as spam. If it is spam it will put the offending thread or reply in the moderator queue.

In order to use Akismet you need to get a API key. You can get one by signing up on WordPress. The Akismet service is free for personal use. Check the Akismet website for more information.

This vBulletin plugin is available under the terms of the GNU Lesser Public License.

Installation instructions are available in the ZIP archive.

Features

• Scan for spam in new threads and post
• Scan content edits
• Automatically put identified spam in the moderation queue
• Report posts as spam or ham (individual post or through inline moderation)
• Limit spam detection to post count and user groups
• Extendable through hooks

Changelog
v1.3

• Added an akismet activity log in the admincp

v1.2

• Fixed an invalid URL issue when using cURL

v1.1

• Option to send parsed post data
• Option to send additional webserver request data for the initial spam check
• Removed product version query
• Added extra information when reporting a post
• Only check API key when entered in the vBulletin options, not for every Akismet request

v1.0

• Various bug fixes: report post submitted wrong data; few missing phrases
• More code documentation
• This plugin now exposes some hooks that others can use to extend the feature set. Like give users infractions when posts are considered spam.

v0.3

• Added options to exclude posts from checking based on user's postcount and usergroups
• Added mass reporting of ham/spam through the inline moderation system.
• Improvements of the internal code

v0.2

• Check spam status on edits
• Add report as ham/spam functionality (requires template edit)

I welcome any feature suggestions and code additions. Feel free to help me improve this mod.

nice to see...hope we'll find it very much usefull

Installed..... :)

Thanks for your efforts.... :up:

I'm so sick of banning and deleting spammers who register and login, starting threads about prescription drug sales or pornographic sites.

I hope this works as I've never handled moderated threads before. I'm not even sure how that function works in vBulletin.

Regards,
Badger

Edit: Question: NM, got it figured out and now have "Quick Moderation" page appearing on Home Page. Thanks.

Ok, version 0.2 has ham and spam reporting.

When you encounter a spam message that was not detected simply press the red flag to submit that message as spam. This will increase the chance that the message will be detected as spam the next time. In the same way you should report a message as ham (green flag) when it was detected as spam. A successful report will automatically mark the post and put it in or remove it from the moderation queue. Of course only forum moderators will be able to report posts as ham or spam.

The red exclamation icon tells you the message has been identified as spam.

In the process of updating to 0.2.... :)

It says we need to modify the template postbit.....

Should we not also modify the template postbit_legacy as well, if boards use it?

Thanks

Regards,
Badger

Since there is no place in this mod that I can find to identify who is a forum moderator or not, can I assume you've defaulted to UserGroup numbers 5,6,7 (Senior Moderator, Admin, Moderator) as the basis for this?

Thanks :)

Regards,
Badger

ps: Can I make a suggestion? Instead of the phrase "ham", why not call it "Not Spam" to avoid confusion?

Er.. yes, if your board uses it you should also update it.

no usergroup check, it simply relies on vBulletin's API. if the user has privileges to moderate posts in a forum.

"ham" is the going name for "not spam", you could simply change the phrases that use "ham".

Done... :)

Yup, tested it various ways logging in as just a normal user and it works fine... thanks.. :)

I didn't know that... must be getting old... ;) I think 'll change the phrases locally... :up:

Nice mod... now I'm just waiting for a spammer to show up so I can see what happens. :D

I assume this doesn't put a performance (or bandwidth consumption) drain on the system when it's checking externally against their database?

Thanks again...

Regards,
Badger

I used to have it installed, its a great modification, thanks for the update.

Depends on what you call a drain. Post will only be checked when they are created or edited. Only the post body (and a few other pieces of information) will be send to the akismet servers. This traffic is much less than an average user viewing the forum.

Starting from 0.3 you can limit the spam checking a bit. E.g. don't check spam for users in certain usergroups and/or check users with less that X posts (defaults to 50). Spammers are usually new users. No real need to check with established users.
50 might be a bit on the high side since it would take the user 50*post delay before he will get around the spam checking. For example if your post delay is 30 seconds it will take the spammer at least 25 minutes before he can start spamming the forum and get away with it. Of course the each post being made must be a valid post.

Nice mod, especially like your improvements!

Great feedback, that's what I suspected about bandwidth and load.... :)

Your 0.3 enhancements (and logic) are right on the money. Spammers on our systems are usually one-timers. They show up, go through the registration process including even doing the email validation, then they post 1 or 2 messages selling porno, drugs or cameras in a couple of public forums. We catch them, ban them and move on.

So, your usergroup check and amount of posts made measurements are exactly how to maximize the mod and minimize the activity.

Thanks for your efforts... :up:

Regards,
Badger

Ok, in .3, I added the additional template change to showthread for in-line moderation, but I can't see anywhere what it does?

Edit: Never mind.... found it... was looking in the wrong place ... very nice design...

Thanks... :)

Regards,
Badger

/me clicks install

Ok, this product is now complete. Everything I planned to add has been added. Other programmers can extend the functionality through the exposed hooks in the code.

Some remarks:

• post content is sent in bbcode format to Akismet; wouldn't it be better to parse it first?
• verifykey is called each time a comment is posted; wouldn't it be sufficient to use verifykey at the time when the key is entered in admincp?
• an additional query is used every time a comment is posted to determine the installed product version; wouldn't it be sufficient to use a define for the product version and to save the query?

Great add-on!

1) I don't know. There is not much info on how akismet really check the content. And spammer often use standard messages for all spam runs, parsing to bbcode could result in different content for each forum.

2) Probably. The documentation isn't very clear on that subject. Then again, if the key is invalid it will probably be notified in the spam check or report.

3) I felt lazy, didn't want to update the define everytime I changed the file. Then again, it's a little effort for a more or less useless query.

Corrected above points in version 1.1

Just installed v1.1 and now I have an error under the AdminCP function....

I have a valid key and haven't touched this field since I original installed the mod and received the key from Wordpress. The error appeared after installing v1.1.

Hope this helps find the bug....

Should I try to do an overwrite and replace v1.1 backwards with v1.0?

Regards,
Badger

Really? I don't get that error for my key. Are you sure that key is correct? Nothing changed to the key validation since 1.0 (except that I removed revalidation during spam checking).

If you have the vBulletin debug functionality enabled you can see the response from akismet in the "debug message" dropdown list

ps, reverting to 1.0 should be possible

Perhaps it doesn't like that change?

I applied for the key when you first released the program. Entered it into the field and haven't touched it since, all through your various upgrades. This error came up as soon as I installed v1.1

I went to the Wordpress web site and validated my key and it's entered correctly. I even cleared the field and started entering it one character at a time as their FAQ said that sometimes cut-n-paste causes problems. About 1/2 way through the error message came up.

Should I uninstall v1.1 and try to re-install, or simple re-install v1.0 over v1.0?

BTW, where do I find the DEBUG drop down list?

Regards,
Badger

Ok, I found how to activate DEBUG mode and at the bottom it says:

Is that what you're looking for?

Regards,
Badger

Ok, reverted back to v1.0 (importing over v1.1) and the error is still there... :(

After that, I completely UN-INSTALLED the mod and re-installed v1.1 and the error is still there. :(

Regards,
Badger

be sure to also overwrite the functions_akismet.php when you upgrade/revert

anyway, there appears to be connections issues, the debug line should be either:
"Key = valid" or "Key = invalid"
if it doesn't contain any text then there was an error in connecting to the akismet server to validate the key.

the key isn't validated while you are typing, it will be validated when the text field loses focus, and when the options pages is loaded.

Besides, you can simply enter an invalid key and save the the options. The rest of the plugin will still try to check for spam and submit content despite a possible invalid key.

Did that ... :)

The message actually pops up during typing...

Perhaps if I sent you a PM and gave you access, you could debug the problem better?

Thanks

Regards,
Badger

Cool! :up:

Version 1.1 has a bug when using cURL for communication (cURL will automatically be used when it's available). For some reason it worked perfectly on my system, but on an other system cURL didn't accept the broken url.
So if you have cURL installed on the system you should upgrade to 1.2, if unsure also upgrade.

Thank you BadgerDog for letting me use your board to figure out this problem.

You're welcome .... :)

Installed v1.2....

All fixed and running smooth.... :up:

Regards,
Badger

Ok, had my first spammer visit since installing this mod.

He posted two threads selling cameras and neither was "moderated" by the mod or "Akismet", so I submitted them to Akismet marking them as spam. This particular ad is quite common and they've visited and posted it before, so I guess the real test now is when they show up the next time, does the Akismet check recognize their next similar post as SPAM

Will advise ... :)

Regards,
Badger

Update....

I had another spammer come back and post the same ad as I reported to Akismet using this mod, as I indicated in the above post.

Unfortunately, the exact same ad (even though reported) still was posted twice in two different forums and was not caught and moderated. :(

So, the conclusion is that something isn't working right, either with the mod itself, or with the interaction using Akismet for checking posts before they actually end up live in a forum.

I have again marked the posts as SPAM and the routine of this mod says it has reported them to Akismet, as well as the mod moved them to become moderated.

Not sure what I should do at this point for testing?

Regards,
Badger

if you want to test if things are working create a user with the name: viagra-test-123
every post made by that users should be qualified as spam.

just to be sure I just check if everything is send correctly to the akismet server, and it is.
they are not very clear on how they detect spam and so far I haven't had a confirmed spam message (or even spam at all).
but if they use a learning system the detection should improve every time you mark content as spam.

but I'm sure the detection will become better as the user base increases, or at least I assume there are major differences between the content of blog spam and forum spam.

anyway, I'll get in contact with the akismet people and try to figure out some more stuff

Thanks for your support ... :)

I'd be interested hearing what they have to say, particularly why an ad with exactly the same content is reported using their format, yet when posted again it doesn't get recognized by their software routines.

That's a good idea about creating a test user called viagra-test-123... :up:

In fact, I have a private forum that I can use to create ads with various obvious spam phrases, so I'll try to create a bunch of new posts using that name as well as put content in them that should be picked up.

I'll let you know the results.... :)

Regards,
Badger

Ok, I created that usename viagra-test-123 and tried a post.....

When I hit submit for the post, the screen cleared and there was a statement that went by so quick I could hardly read it. Something to the effect of my post was being submitted for moderation. They really should halt that "screen" with a button for OK to continue so people can see what it says. :D

Anyway, apparently it didn't like the post from username viagra-test-123 and did immediately put it into moderation que ... :up:

Here's the screen that I saw when I went to the que and checked it.

So, I guess the word viagara in the username was the catch phrase here, since nothing else in the subject or body of the post (except the word spam) was threatening enough to trigger a moderation event.

I'm going to do some more testing using a username that's not obvious and try to put some various phrases into the body of the posts that are obvious. ie: viagara.

I suspect that will work, but the real issue should be that once I've reported something as spam from my user site (web address), I would have thought that Akismet recognizes it going forward as spam, at least when associated with my site alone.

That's what I think it's NOT doing, because HUGE ads for cameras and cell phones being reported doesn't seem to be enough to get the same ads on their list.

Thanks again for your work. :)

Regards,
Badger

Update again ..... :)

Ok, using the username Tester this time, I posted message with the phrase viagara in the BODY of the post and also created a second post with the phrase viagara in the SUBJECT header.

Neither one was picked up by Akismet.....

So, I guess it picks up viagara when part of the username, but nowhere else?

Hope this helps....

Regards,
Badger

Edit: Out of curiosity, I set the option "Send extra request information" to ON and tried the above tests in this post again, however, it made no difference and Akismet failed to spot it.

The username 'viagra-test-123' is a testing feature, akismet will consider everything from viagra-test-123 as spam. It's just for testing purposes to check if the system is still properly communicating with the servers.

Anyway, just uploaded 1.3 which adds an activity log for akismet actions.

V1.3 installed and functioning..... :up:

I can see new log function at bottom of "Statistics and Logs" area of AdminCp....

Will advise results....

Thanks... :)

Regards,
Badger

Ok, serious issue..... :D

Just tried to post simple test message under username Tester and when I hit submit, got a white Vbulletin screen saying DATABASE ERROR with no other information other then the standard "Go Back" kind of thing.

When I back up, the message never got posted anywhere (even moderation) and I suspct it has something to do with trying to write to the database log?

Since I'm on a LIVE system, I've had to disable v1.3 version of mod in the AdminCP until you get a chance to look at it. I re-tested posting after I disabled the mod and the user Tester was able to post the exact same message with no error, so I assume by leaving the mod disabled the system will continue to operate without problems (although no Akismet checking).

Regards,
Badger

Edit: By the way, there were NO ENTRIES in the log at any time during this testing process, if that helps....

oh crap, forgot to include the table prefix in the install query to create the table. get 1.3.1

Installed v1.3.1 (Overwrite) .... :up:

Re-enabled mod...

Ran same simple post test with username tester as described above...

Got the exact same vBulletin database error.... nothing recorded in log...

Disabled mod and retested..... no error....

Hope this helps track down the databse error problem that began with adding the "logging" function...

Regards,
Badger