Administrative and Maintenance Tools - Auto Bounce Messages Management

Brought to you by Syrian Medical Society & Milado

Introduction: quoted from wikipedia with adaptation.
Any long-lived forums is going to eventually contain addresses that can't be reached. Addresses that were once valid can become unusable because the person receiving the mail there has switched to a different provider (possibly as a result of changing jobs or schools). In another scenario, the address may still exist but be abandoned, with unread mail accumulating until there is not enough room left to accept any more.

The presence of invalid addresses in the forums results in bounce messages being sent to the owner of the forums. If the mailing list is small, the owner can read the bounce messages and manually remove the invalid addresses from the forums. With a larger forums, this is a tedious, unpleasant job, so it is desirable to automate the process.

In short: This hack automates handling of the bounce messages.


Requirements:

• vBulletin 3.7.1
• catch-all email address, with POP3 access. (I use Google app (Gmail))

Features:

• Auto Bounce Messages Management.
• Uses POP3 protocol to retrieve messages from your catch-all email box.
• On bounce detection the user will be moved to bounce usergroup (you can set which usergroup).
• Works according to VERP.

A note:
I believe there is no need to delete the user subscriptions. because once a user got moved to the "Awaiting email confirmation" usergroup, he / she won't be sent any additional emails.

How Does It Work?
You have your catch-all email box, that will be used to receive the bounce messages. Of course we must have a POP3 access to retrieve messages from it.
Let's suppose that your catch-all email address is bounces@yourdomain.ext
After we set it properly, it will work as described below:
1) By default: On every single message being sent, vBulletin generates a unique Message-ID for it, example:

Message-ID: <20080608103128.551e45224330@www.yourdomain.ext>

This ID is generated By using the following code in /includes/class_mail.php:

PHP Code:

$msgid = '<' . gmdate('YmdHis') . '.' . substr(md5($message . microtime()), 0, 6) . vbrand(100000, 999999) . '@' . $http_host . '>'; 


I take the red portion from the unique Message-ID, and the red portion from the domain name of the catch-all address:

Message-ID: <20080608103128.551e45224330@www.yourdomain.ext>

bounces@yourdomain.ext

The result will be:

bounce20080608103128.551e45224330@yourdomain.ext

Now every single message has its own unique envelope-from address. so if it gets returned as a bounce, I will recognize it easily.
So every message will be recorded in the database. each record consists of : toemail, bounceemail (the one we generated) and the timeline (for sake of pruning.)
2) A cron job runs every ten minutes to retrieve bounce messages from the catch-all email box.
If the message sent to something similar to the bounceemail we generate, it will update the appropriate record in the database to be marked as returned. and delete the message from the POP3 server.
3) Another cron job runs once an hour, reads the bounced emails from the database and moves their owners to an usergroup which is determined by the admin from the settings.
4) Yet another cron job, will run once daily, to prune the records that haven't returned bounces in a settable period of time (30 days by default).

Did I miss something here?
Yeh, right. this is the first beta, I believe the core is almost done. but there may be some functions to add, and few things to tweak.
I'm all opened to your suggestions. So feel free to dream!

Installation & Upgrade:
1) Have your own catch-all email with POP3 access. I don't have the time to write a how-to article for this please do it yourself, you'll be luckily if someone explains it to others.
2) Upload files from the /upload/ folder in the package to their appropriate locations.
3) Install the xml product file: product-bounce_management.xml, of course choose yes to override when upgrading.
4) Refresh your AdminCP. you will see at the navigation side, above the Users tab, a new tab called "Bounces Management"
5) At that tab, go to the settings: Set your POP server settings, your "Bounce Usergroup" and "Exclude Usergroups".
6) At the same tab you will find a link called "Test POP Connection" to test the connection with your POP server.
7) Enjoy! Click install to let me enjoy too!

Settings:
Attachment 81933

Know issues:
- In rare situations, a vacation reply may returned via the bounce path rather than the webmaster path.


Change log:

• 1.0.0.b.1 first public release, on June, 8 2008.

Support:
The product marked as not supported, but that doesn't mean you won't see me again here.
So please try to help each other.

License & Credits:
1) I use the pop3_class from Manuel Lemos, Copyright (c) 1999-2005 to retrieve messages from the POP server, so all the credits regarding this go to him.
Here is his license (included in the package too):

Copyright (c) 1999-2005, Manuel Lemos
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of the Manuel Lemos nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Click install to receive updates.
You like it? nominate it. I'm looking forward to enter the Mod Of The Month competition.
I depend on you.
Also donations will help me to improve the product. :up:

[Milad]

Raptor, please don't push!

[Gene Steinberg]

Originally Posted by Gersfan

I guess It's not working them, I'll probarly have to install the EZ version and manualy move accounts.

I can't get the EZ version to function at all with 3.7.2. I get the bounced mails, but they don't have the requisite links that EZ is supposed to insert. I tried both the header and the insertion at the bottom of the message -- no luck. The coder doesn't respond to the post I presented on the subject.

So I'm back to this one, but it's not fully reliable either. It doesn't appear to be parsing even a fraction of the actual bounces.

Milad, any help here?

Peace,
Gene

[Milad]

This is beta 1, you're expecting too much from the first beta.

I worked on beta 2. but din't finish it yet. Some commercial projects have higher priorities.

So please be patient.

[Gene Steinberg]

Originally Posted by Milad

This is beta 1, you're expecting too much from the first beta.

I worked on beta 2. but din't finish it yet. Some commercial projects have higher priorities.

So please be patient.

I just want it to work.

Do you have any ideas yet when beta 2 will be finished?

Peace,
Gene

[fatal1980]

Originally Posted by ThorstenA

I agree that a copyright notice is okay. But I need to know that there is a notice and where it is placed. Here it is placed on *all* of my pages. This for an administorial modification! You'd never think there's a copyright notice at all. If administrators just install this mod, they do not always look around what this modification displays on places which have nothing to do with this mod.

I never installed a mod and then searched my website for additional copyright notices (which is here also linked - bad for SEO). And I installed a lot of mods. But now I do. Does not make vb.org a more secure place for me.

Deal with it. He isn't Hans Reiser, he didn't murder his wife. So don't get so bent out of shape.

[fatal1980]

I do have one suggestion. If a message is going to bounce properly, the messages sent from vBulletin wouldn't even be accepted by the destination host. Typically if an email no longer exists, the message is bounced during the SMTP transaction.

Instead of your current method of verifying bounce messages. Why don't you either re-write the vBulletin sending function or place a hook before the message is to be sent. And have a script resolve the domains mx record. Then connect to the destination mail server and confirm the message is valid. You pretty much just have to issue a HELO, MAIL FROM, and RCPT TO. You don't even have to send a message through.

This definitely cut down the time and resources used to determine if an address is legitimate. You would obviously have to setup the pop account as well to catch mail servers that are accepting mail and then bouncing it (which is bad and shouldn't happen).

Furthermore, the process of the bad mail addresses would happen during regular mail operations.

Thanks for the great mod.

[Milad]

Originally Posted by fatal1980

I do have one suggestion. If a message is going to bounce properly, the messages sent from vBulletin wouldn't even be accepted by the destination host. Typically if an email no longer exists, the message is bounced during the SMTP transaction.

Instead of your current method of verifying bounce messages. Why don't you either re-write the vBulletin sending function or place a hook before the message is to be sent. And have a script resolve the domains mx record. Then connect to the destination mail server and confirm the message is valid. You pretty much just have to issue a HELO, MAIL FROM, and RCPT TO. You don't even have to send a message through.

This definitely cut down the time and resources used to determine if an address is legitimate. You would obviously have to setup the pop account as well to catch mail servers that are accepting mail and then bouncing it (which is bad and shouldn't happen).

Furthermore, the process of the bad mail addresses would happen during regular mail operations.

Thanks for the great mod.

That's bad approach. you don't need to send a "Hello" email whenever you want to send an email.

[fatal1980]

Originally Posted by Milad

That's bad approach. you don't need to send a "Hello" email whenever you want to send an email.

You dont send a "HELLO" email message. You just connect to the SMTP server for the domain name and send some regular SMTP commands to verify if the email exists.

Ill show you how it can be done manually.

Find the MX record

[comm@taco:/home/comm]> dig hotmail.com mx

;; ANSWER SECTION:
hotmail.com. 3600 IN MX 5 mx4.hotmail.com.
hotmail.com. 3600 IN MX 5 mx1.hotmail.com.
hotmail.com. 3600 IN MX 5 mx3.hotmail.com.
hotmail.com. 3600 IN MX 5 mx2.hotmail.com.

Connect the the one of the MX records (I've *** out my email address):

[comm@taco:/home/comm]> telnet mx2.hotmail.com 25
Trying 65.54.245.40...
Connected to mx2.hotmail.com.
Escape character is '^]'.
220 bay0-mc10-f11.bay0.hotmail.com Sending unsolicited commercial or bulk e-mail to Microsoft's computer network is prohibited. Other restrictions are found at http://privacy.msn.com/Anti-spam/. Violations will result in use of equipment located in California and other states. Fri, 22 Aug 2008 19:53:02 -0700
HELO test.example.com
250 bay0-mc10-f11.bay0.hotmail.com (3.6.0.91) Hello [64.40.106.55]
MAIL FROM: j***@gmail.com
250 j****@gmail.com....Sender OK
RCPT TO: n*****@hotmail.com
250 n****@hotmail.com
DATA
354 Start mail input; end with <CRLF>.<CRLF>
^]
telnet> Connection closed.

As you can see the mail was accepted after the DATA command. So this was successful.

Now lets see one thats failed on an address that didn't exist.

[comm@taco:/home/comm]> telnet mx2.hotmail.com 25
Trying 65.54.244.168...
Connected to mx2.hotmail.com.
Escape character is '^]'.
220 bay0-mc6-f8.bay0.hotmail.com Sending unsolicited commercial or bulk e-mail to Microsoft's computer network is prohibited. Other restrictions are found at http://privacy.msn.com/Anti-spam/. Violations will result in use of equipment located in California and other states. Fri, 22 Aug 2008 19:56:44 -0700
helo test.example.com
250 bay0-mc6-f8.bay0.hotmail.com (3.6.0.91) Hello [64.40.106.55]
MAIL FROM: ****@gmail.com
250 *****@gmail.com....Sender OK
RCPT TO: badmail1231233123@hotmail.com
550 Requested action not taken: mailbox unavailable
^]
telnet> Connection closed.

As you can see the SMTP session fails! Easy test and you don't need to send a message to the server or process bounce messages. This is the same process used in the following module that works very well:

https://vborg.vbsupport.ru/showthread.php?t=144590

Your wasting your time and resources on use a POP3 account, just use the above method. While your at it you can also re-create the above module for 3.7 since it hasn't been coded over to 3.7 and it would be fairly easy.

[Gene Steinberg]

I don't presume to know why this particular product doesn't work, and I'l accept your logic. For now, I'm using EZ Bounce which, while requiring some manual labor (and a plugin mod until the coder added it himself), does what it supposed to do. And it works with 3.7.2PL2.

Peace,
Gene

[TooTall52]

This won't work accurately as many email servers out there accept all mail, process it then kick back bounces for the invalid ones. microsoft exchange works this way for example, may Lotus Notes as well.

so just because a helo passes does not mean it is valid address, the seperate pop account is probably more accurate.

TooTall52

Originally Posted by fatal1980

You dont send a "HELLO" email message. You just connect to the SMTP server for the domain name and send some regular SMTP commands to verify if the email exists.

Ill show you how it can be done manually.

Find the MX record

[comm@taco:/home/comm]> dig hotmail.com mx

;; ANSWER SECTION:
hotmail.com. 3600 IN MX 5 mx4.hotmail.com.
hotmail.com. 3600 IN MX 5 mx1.hotmail.com.
hotmail.com. 3600 IN MX 5 mx3.hotmail.com.
hotmail.com. 3600 IN MX 5 mx2.hotmail.com.

Connect the the one of the MX records (I've *** out my email address):

[comm@taco:/home/comm]> telnet mx2.hotmail.com 25
Trying 65.54.245.40...
Connected to mx2.hotmail.com.
Escape character is '^]'.
220 bay0-mc10-f11.bay0.hotmail.com Sending unsolicited commercial or bulk e-mail to Microsoft's computer network is prohibited. Other restrictions are found at http://privacy.msn.com/Anti-spam/. Violations will result in use of equipment located in California and other states. Fri, 22 Aug 2008 19:53:02 -0700
HELO test.example.com
250 bay0-mc10-f11.bay0.hotmail.com (3.6.0.91) Hello [64.40.106.55]
MAIL FROM: j***@gmail.com
250 j****@gmail.com....Sender OK
RCPT TO: n*****@hotmail.com
250 n****@hotmail.com
DATA
354 Start mail input; end with <CRLF>.<CRLF>
^]
telnet> Connection closed.

As you can see the mail was accepted after the DATA command. So this was successful.

Now lets see one thats failed on an address that didn't exist.

[comm@taco:/home/comm]> telnet mx2.hotmail.com 25
Trying 65.54.244.168...
Connected to mx2.hotmail.com.
Escape character is '^]'.
220 bay0-mc6-f8.bay0.hotmail.com Sending unsolicited commercial or bulk e-mail to Microsoft's computer network is prohibited. Other restrictions are found at http://privacy.msn.com/Anti-spam/. Violations will result in use of equipment located in California and other states. Fri, 22 Aug 2008 19:56:44 -0700
helo test.example.com
250 bay0-mc6-f8.bay0.hotmail.com (3.6.0.91) Hello [64.40.106.55]
MAIL FROM: ****@gmail.com
250 *****@gmail.com....Sender OK
RCPT TO: badmail1231233123@hotmail.com
550 Requested action not taken: mailbox unavailable
^]
telnet> Connection closed.

As you can see the SMTP session fails! Easy test and you don't need to send a message to the server or process bounce messages. This is the same process used in the following module that works very well:

https://vborg.vbsupport.ru/showthread.php?t=144590

Your wasting your time and resources on use a POP3 account, just use the above method. While your at it you can also re-create the above module for 3.7 since it hasn't been coded over to 3.7 and it would be fairly easy.