Ill see how I can add these options. If they were good during registration as a sleeping spammer, we would have to test their details again. This could be done the first time that they post or send a PM. I dont think that would be too hard to add. Ill look up the hooks at add that to the list of things to do.

Contacts being Contact Us I guess? Using the above method, I could do that. You would need to change the blocking text from "If you think this block is in error, use Contact Us" to "hmm, no change Charlie" :)

Sorry, I should have been more clear.

1) These would be enabled / disabled by admin preference

2) The notion would be that it would only check posts/pms for users that belong to specified user groups. This supports the notion of using promotions on your site to identify trusted users (ie. registered users versus trusted registered users (after 6 days and 3 posts or whatever your flavor is)

3) Contact, yes Contact Us. The understanding being that if you enable this you have prevented users from contacting you if they are incorrectly banned. Mitigation could include javascript, image based or some other obfuscated version of your email in the failure message)

wow, Im flattered, thanks. That would be great even to be nominated for the vote.

2 - There is an option in 0.7 to put new users into a specific group. You could use that to apply filters against. Ill have to up the database cache time incase Russ at sfs.com gets hammered/DDoSed by large forums that leave users in a group like this. Maybe I could include a "cut off" period for users in this group, where it doesnt check them if registered over 30 days (or X) days ago.

What I could also do is add a cron job to pull the daily IP ban list and put that into the cache. They should catch a lot of spam without the necessary remote queries. Its a fine balance between protecting against spam and DDoSing the limited resources of a one-man free website :)

If Russ was to move the lot into DNS instead of XML/HTML, that would make things a lot easier as he would have DNS servers caching data :)

3 - I believe captcha can be enabled on the ContactUS page but as the loading would be much less on this page than people posting into forums, I dont think lookups would be a problem here.

Captchas can be enabled (as can custom questions on 3.7).. sadly It seems that I am dealing with human spammers (mix of china, india, russia and us based) <sigh>

well, Ill add a IP based lookup for the Contact US page, pulled initially from the dialy cron job that pulls the IP lists and failing that, from a live lookup.

Voted as one way of thanks!!

Yes, voted, too - before the 1st reminder post ;)

Im flattered, I really am.

Update on 0.7rc.

The core caching / whitelist / lookup rewrite is complete with the following requests included
- GeoIP country banning (goodbye China - nothing personal but you spam too much)
- Much better caching support
- Better remote lookup support
- Statistics logging
- Locally cached cron import of the stopforumspam IP database

The core has been rewritten to allow easy integration into the Contact Us form and into post thread / message or PM so that you can configure it to check if details appear on the database for X days after registrations. This will allow a system admin to set, for example, a threshold of 30 days. If posting a PM or message within the first 30 after registrations, it will check the database if they appear as a spammer and will block it, with an optional PM notification to a mod/admin group.

Also included in the starter of the UI support, is full 3.6 and 3.7 integration, giving details of whois information, links to google searches, etc etc.

With the statistics module (that Ive started), youll get full graphs on a yearly period (older than a year gets auto pruned), with a daily, monthly and yearly totals/averages of
- Cache hits vs misses
- Country ban breakdowns
- Successful vs Banned vs Expired* registrations

*Expired is where there is a hit on the spam database but its older than the threshold set in the forum.

Installed and testing ... :up:

Thanks .... :)

One thing that has me a little confused is the field for:

Is this something I need to use this mod effectively and if so, how do I obtain a key?

Appreciate your efforts in trying to help us block spammers...

Regards,
Badger

the API key is available from www.stopforumspam.com - its a feature that never quite made it into 0.6 but will make its appearance in 0.7. It allows you to submit user details to the ban list on stopforumspam.com

Thanks pedigree.... :)

Is there any way to test this mod after install to make sure that people can still register and it's not blocking legitimate users?

I installed it earlier, but I haven't seen any new registrations since then, so I wanted to make sure everything was functioning ok.

Looking forward to v.07 .....

Regards,
Badger

You can disable blocking in the options and set actions to "Log and allow registration".

if you look at admincp -> logs -> vbstopforumspam -> submit

you should see if youve have any new registrations logged by vbsfs

Im sitting here look at all the debug logs for 0.7 and I might go to bed now :D

Badger,

In theory, you can register for your own site to test, and then test by registering while browsing your site with a few random anonymous proxy websites (http://www.the-cloak.com , etc.)

Thanks ... :up:

I did a test by pretending to be a spammer and using an email address that was known from the "stop forum spam" web site and that seemed to work fine. I got the rejected error message and a log entry was created. Unfortunately, it was from my own IP address, so I hope it didn't send that anywhere reporting my IP address as a spammer.. :D

I'll have to try registering using correct information and see if it lets me though now ...

Regards,
Badger

0.6.1 does NOT send your info anywhere to be banned. You'll be able to log on just fine.

Thanks .... :up:

I've tested it thoroughly now and it seems to be working fine. I guess the real value of this is how accurate and up to date we can keep "stop forum spam" using the feedback mechanism of this mod, once it's available in v0.7.

Appreciate the author's efforts in trying to help us all deal with spam. It's the most annoying aspect of trying to run a professional and mature Vbulletin environment.

Regards,
Badger

Thanks for the feedback all.

A couple of people that have added to comments and provided feedback are running larger boards and havent had any issues with 0.60/0.61. It seems stable and is working well, not as well as it could be but that will be addressed in the next version.

What Wired1 said about sending information, it doesnt. Its completely passive and only ever retrieves information. 0.7 will have the functionality to submit details based on group membership. I hope Russ is updating sfs.com in real time but he seems to be a very difficult person to get a response out of.

Installed on vb 3.7.1 pl2 with php 5.2.6 and curl. I will let you know how it goes.

Many thanks for this!!

BTW: does this mod extend the time needed for registration? Does it conflict with ISBOT?

You're welcome.... :)

Thank you for investing your time in this... appreciate it... :up:

Been using it for about a day now and the log is building. I notice one set of entries that repeated 6 times with the last one (2008-07-01 09:56:33) having the underline clickable link,whereas the other entries don't.

Why would this repeat that many times?

BTW, I put in X's and chanhed the username to protect the registrant's privacy.

Regards,
Badger

Badger hmm thats strange, it might happen I guess if the user refreshed the page and then has data resubmitted. What Ill do is add code to check if the user/email is in use and them stops processing, allow control to pass back to vbulletin for rejection... No point doing all that processing if the user exists.

Thanks for that, it made me thinks

And this is an open request...

Post here what you would like to see in the ways of stats, charts, graphs etc...

so far Ive got

Bar graphs
- Registrations per hour blocked / allowed by policy / allowed with old data / allowed clean / whitelisted / blacklisted

Pie charts
- Blocked domains by domain name
- Blocked domains by country
- Blocked registrations by domains
- Blocked registrations by IP in 16/24 bit subnets (hard on the CPU)
- Cache hits vs misses vs time

Hi pedigree ... :)

I've been using this mod for the last 6 months without any problems, but since it handles the registrattion page differently thatn the default vbulletin code, could that be why it repeated 6 times? If he kept making mistakes and it's a java code based registration, it might be a factor?

https://vborg.vbsupport.ru/showthread.php?t=144869

Regards,
Badger

Nope. When they hit submit, it compares their IP / email address / user name against a list, and if they're on the list, no entry permitted. I believe they just come back to the registration page so they can change their user name / email address.


pedigree: in 0.7.0, can we now sort the log by all columns, and/or do a search on it? As a secondary suggestion (and I can see this somewhat being outside the scope of your mod, so if you agree, no biggie), maybe compare all people who were banned in the past month, and compare it to successful registrants in the vBStopForumSpam logs? This way, an admin can have a quick list to manually report to the black list (e.g. hitting a button to do it)? This would allow the admin to only report those who have banned because of spamming, and not violating other rules on the forum.

Come to think of it, if someone was PERM banned via an infraction (granted not many use infractions in this way I think), maybe this mod could somehow detect it and auto-report it?


Badger, the repeats are normal. It's the user screwing up, that's it. I see it all the time in my logs, and my register form is standard. They're just mis-typing or something.

Now, if the error was added to the log, that would be nice, although I'm not sure how much it would help to prevent spam.

Took a bit of time and tallied up stats for all of June for one of my sites

Out of the 7180 registrations for the 30 day period:

~.03%/250 were caught by SFS Email check
~25%/1860 were caught by SFS UserID check
~45%/3260 were caught by SFS IP check

~25%/1810 made it past SFS Check

note: of the 1810 that made it past SFS, 780 of those were caught by RBL Checking and rejected, 12 slipped through completely but were manually discovered, no false positives that I am aware of.

How is the data being aggregated? Have you indexed the log table for reporting or are you normalizing the reporting data elsewhere?

I'd be interested in some long term % based trending (week, month, year) as line graphs

Things like:

% of blocked registrations over time
Of blocked registrations, % email, % ip, % username on a single graph over time

Thanks again and Great Job!

At the moment, the only data that I have going to a stats table is cache hit/miss numbers, records on an hourly total. Ive done that with a roundrobin type system, almost. It purges records over 1 year old (just as RRDtool does) and logs on hourly totals, inserting new records and updating existing ones (for those with mysql knowledge, its an insert on duplicate key update with an index on the primary date field)

I thought about using RRDtool but couldnt find a pure php implementation of it as I couldnt rely on hosts having binary execution rights etc.

All of the graphs are going to have weekly/monthly and yearly trending for each - something that I hope wont kill the server too much, hence why Im using hourly totals rather than just inserting a new row per sample.

The stats tables will be seperate from the log data so if you purge your logs, the stats will remain untouched.

There is also a diagnostics logs, off by default, where it will attempt to log each step of the registration process with records purged after 7 days. As this table contains a TEXT field, I wanted to keep the size down. This is mainly for debug review if something starts going wrong with the code.

skippy - I was thinking about RBL checking but there is already a package doing so that I left it at that :) Would you like to see something like it in this, with graphing? I was thinking about adding SURBL support in a later version.

Wired1 - You will be able to sort on all the fields and Ill add a search as well. It will be a match on any field, you wont be able to specify which field you want to search on as I want to get this released before my son is born. Ill be working during my paternity leave when I really shouldnt, so I wont have too much spare time. As for the ban/report, its a bit out of the scope but Ill add it to the list for future releases.

pedigree, a single mod to manage all of my spam validation would be a holy grail for me...

that being said, having two right now is not the end of the world and many things on your list seem much higher priority (especially that whole son thing :-p)

It seems to work very well on vb 3.7.1 PL2

So well, that I am being flooding by emails from this hack. How do I turn them off?

Ok, I caught a spammer who wasn't blocked by "stop forum spam" site.

How do I now get that information, IP address, email address and user name to their site?

Thanks... :)

Regards,
Badger

@Alpha1: What emails are you receiving? I don't remember there even being an option for this mod to email?

@BadgerDog: You can add new entries at: http://www.stopforumspam.com/add

Thanks... did that ... :up:

My first "kill"... :D

Regards,
Badger

Good to hear about 3.7PL2.

Are the emails coming from the "Contact Us" page? If so, then two ways really

1. Edit the phrase and remove the "Contact Us if you think this is wrong" incase they didnt think of doing that

2. Disable the "Contact Us" page in vBulletin options.

Are the registrations all spammers? Maybe you could PM me a copy/paste of your logs so that I can verify theyre spammers and not the plugin gone mad.

Edit : This mod doesnt send emails

.. oh, the next version will do spambot filtering on the Contact Us page as well :)

This looks awesome - and *exactly* what I need at the moment! :) My forum gets around 55,000 unique visitors and 250,000 pageviews a day - and it's being ruined by spammers! I'll post back in a few days to let you know the results (it's already caught one and I only installed it 60 seconds ago!).

Thanks pedigree! :)

Ok, I just had a legitimate new member try to register and he got the spammer rejection message (our very first)....

It didn't like his IP address, indicating it has been used by someone to spam forums in the past. Here?s the information from the log:

What's very odd is that the actual IP address he was using when he tried to log on our site was IP Address: 195.93.21.8, which differs from the one he was rejected for?

Is something broken here? :confused:

By the way, I changed the username and email address for pivacy purposes.

Regards,
Badger

Checking that IP on the stopforumspam.com website, it's linked to an AOL.com email address. AOL also owns that IP range. Most likely someone was spamming from a computer w/ AOL as the ISP. AOL frequently cycles IP addresses around, so that's how they got that IP.

AOL operate transparent proxy servers that filter/monitor and record all HTML data passing from their networks, for many reasons, data mining, advertising etc... It being on the same 24 subnet is a bit strange bit who knows what AOL think.

ISPs that transparently redirect/filter and cache HTTP traffic are always going to be a problem when filtering on IP numbers. I thought about filtering on the HTTP_FORWARDED_FOR headers fromthe caches but that would easily allow a spammer to inject headers and therefore bypass filtering.

All I can suggest is create him an account and mail him the password or decrease the age expiry to under 30 days, that will treat the record at sfs.com as too old and will ignore it.... Or even better, tell him to get his ISP to turn off proxying. A friend owns an ISP and with bandwidth being so cheap compared to the overhead of running caching servers, given the low hits, he turned them all off.

This raises the question of one of the missing features of the site regarding modding up an IP, email or username. Right now you can flag something bad, but can never rectify that (unless you flagged it and manually remove it from your list of spammers)..

Obviously a concept like this is prime for fraud if not done correctly.

Anyone have any suggestions on how we may implement a "whitelist" feature without inviting automated bots to pollute the database?

I think those questions are best raised on the stopfrorumspam.com website. I did kind of foresee that and thats why I put a "lastseen" expiry in the mod. By default, from memory, its 90 days. 0.7 has whitelisting features for username/email and IP addresses/subnets as well as the said expiry.

Expiry being that a postive spambot result from stopforumspam.com with a lastseen thats older than the threshold set in your options, will result in a negative spambot result. This kind of works around some of the issues.

That's how I handled this individual...

Thanks ... :)

Regards,
Badger

No solution is ever going to be perfect :(