Administrative and Maintenance Tools - Auto Bounce Messages Management

Brought to you by Syrian Medical Society & Milado

Introduction: quoted from wikipedia with adaptation.
Any long-lived forums is going to eventually contain addresses that can't be reached. Addresses that were once valid can become unusable because the person receiving the mail there has switched to a different provider (possibly as a result of changing jobs or schools). In another scenario, the address may still exist but be abandoned, with unread mail accumulating until there is not enough room left to accept any more.

The presence of invalid addresses in the forums results in bounce messages being sent to the owner of the forums. If the mailing list is small, the owner can read the bounce messages and manually remove the invalid addresses from the forums. With a larger forums, this is a tedious, unpleasant job, so it is desirable to automate the process.

In short: This hack automates handling of the bounce messages.


Requirements:

• vBulletin 3.7.1
• catch-all email address, with POP3 access. (I use Google app (Gmail))

Features:

• Auto Bounce Messages Management.
• Uses POP3 protocol to retrieve messages from your catch-all email box.
• On bounce detection the user will be moved to bounce usergroup (you can set which usergroup).
• Works according to VERP.

A note:
I believe there is no need to delete the user subscriptions. because once a user got moved to the "Awaiting email confirmation" usergroup, he / she won't be sent any additional emails.

How Does It Work?
You have your catch-all email box, that will be used to receive the bounce messages. Of course we must have a POP3 access to retrieve messages from it.
Let's suppose that your catch-all email address is bounces@yourdomain.ext
After we set it properly, it will work as described below:
1) By default: On every single message being sent, vBulletin generates a unique Message-ID for it, example:

Message-ID: <20080608103128.551e45224330@www.yourdomain.ext>

This ID is generated By using the following code in /includes/class_mail.php:

PHP Code:

$msgid = '<' . gmdate('YmdHis') . '.' . substr(md5($message . microtime()), 0, 6) . vbrand(100000, 999999) . '@' . $http_host . '>'; 


I take the red portion from the unique Message-ID, and the red portion from the domain name of the catch-all address:

Message-ID: <20080608103128.551e45224330@www.yourdomain.ext>

bounces@yourdomain.ext

The result will be:

bounce20080608103128.551e45224330@yourdomain.ext

Now every single message has its own unique envelope-from address. so if it gets returned as a bounce, I will recognize it easily.
So every message will be recorded in the database. each record consists of : toemail, bounceemail (the one we generated) and the timeline (for sake of pruning.)
2) A cron job runs every ten minutes to retrieve bounce messages from the catch-all email box.
If the message sent to something similar to the bounceemail we generate, it will update the appropriate record in the database to be marked as returned. and delete the message from the POP3 server.
3) Another cron job runs once an hour, reads the bounced emails from the database and moves their owners to an usergroup which is determined by the admin from the settings.
4) Yet another cron job, will run once daily, to prune the records that haven't returned bounces in a settable period of time (30 days by default).

Did I miss something here?
Yeh, right. this is the first beta, I believe the core is almost done. but there may be some functions to add, and few things to tweak.
I'm all opened to your suggestions. So feel free to dream!

Installation & Upgrade:
1) Have your own catch-all email with POP3 access. I don't have the time to write a how-to article for this please do it yourself, you'll be luckily if someone explains it to others.
2) Upload files from the /upload/ folder in the package to their appropriate locations.
3) Install the xml product file: product-bounce_management.xml, of course choose yes to override when upgrading.
4) Refresh your AdminCP. you will see at the navigation side, above the Users tab, a new tab called "Bounces Management"
5) At that tab, go to the settings: Set your POP server settings, your "Bounce Usergroup" and "Exclude Usergroups".
6) At the same tab you will find a link called "Test POP Connection" to test the connection with your POP server.
7) Enjoy! Click install to let me enjoy too!

Settings:
Attachment 81933

Know issues:
- In rare situations, a vacation reply may returned via the bounce path rather than the webmaster path.


Change log:

• 1.0.0.b.1 first public release, on June, 8 2008.

Support:
The product marked as not supported, but that doesn't mean you won't see me again here.
So please try to help each other.

License & Credits:
1) I use the pop3_class from Manuel Lemos, Copyright (c) 1999-2005 to retrieve messages from the POP server, so all the credits regarding this go to him.
Here is his license (included in the package too):

Copyright (c) 1999-2005, Manuel Lemos
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of the Manuel Lemos nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Click install to receive updates.
You like it? nominate it. I'm looking forward to enter the Mod Of The Month competition.
I depend on you.
Also donations will help me to improve the product. :up:

[J3rico]

hi, thank's for the answer, i have the cron retrievebounces.php (this file is included in the zip file), when run i get this message :

ABMM: Retrive Bounces
Connected to the POP3 server "pop.gmail.com".
User "myuser@gmail.com" logged in.
There are no new bounces!
Disconnected from the POP3 server "pop.gmail.com".

Eseguito

thank's

[Milad]

If the bounce messages are still in your gmail account, please open one of them and look for the "Show original" link in the drop-down menu at the upper right corner.

Please copy and paste the content here.

Regrads.

[J3rico]

Delivered-To: myaccount@gmail.com
Received: by 10.141.20.5 with SMTP id x5cs125266rvi;
        Thu, 19 Jun 2008 10:55:10 -0700 (PDT)
Received: by 10.141.151.18 with SMTP id d18mr6810126rvo.152.1213898110434;
        Thu, 19 Jun 2008 10:55:10 -0700 (PDT)
Received-SPF: softfail (google.com: domain of transitioning unknown does not designate 65.54.246.91 as permitted sender) client-ip=65.54.246.91;
Received: by 10.140.83.37 with POP3 id g37mf110384rvb.6;
        Thu, 19 Jun 2008 10:55:10 -0700 (PDT)
X-Gmail-Fetch-Info: postmaster@mysite.it 1 mail.mysite.it 110 postmaster@mysyte.it
Return-Path: <>

X-Spam-Checker-Version: SpamAssassin 3.1.7-deb (2006-10-05) on IT20-MAIL
X-Spam-Level: ***
X-Spam-Status: No, score=3.0 required=5.5 tests=BAYES_00,DATE_IN_PAST_06_12,
	FORGED_HOTMAIL_RCVD,FRONTPAGE,HTML_IMAGE_RATIO_02,HTML_MESSAGE,
	HTML_TAG_BALANCE_BODY,NO_REAL_NAME autolearn=no version=3.1.7-deb
Delivered-To: mysyte.it-postmaster@mysyte.it
Received: (qmail 14689 invoked by uid 0); 19 Jun 2008 19:45:01 +0200
Received: from unknown (HELO bay0-omc1-s19.bay0.hotmail.com) (65.54.246.91)
  by xxx.xxxxxxxx.net with SMTP; 19 Jun 2008 19:45:01 +0200
Received: from BAY106-W35 ([65.54.161.135]) by bay0-omc1-s19.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Thu, 19 Jun 2008 10:44:56 -0700
Message-ID: <BAY106-W35453E691116F447678B16BAAA0@phx.gbl>
From: postmaster@mail.hotmail.com
To: postmaster@mysyte.it
Date: Thu, 19 Jun 2008 10:44:56 GMT
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
	boundary="9B095B5ADSN=_98C9A2A81C354B9EB49533476B1?WOM1.labinte"
X-DSNContext: 7ce717b1 - 1196 - 00000002 - 00000000
Subject: Delivery Status Notification (Failure)
Return-Path: <>
X-OriginalArrivalTime: 19 Jun 2008 17:44:56.0485 (UTC) FILETIME=[2FF61D50:01C8D234]

This is a MIME-formatted message.  
Portions of this message may be unreadable without a MIME-capable mail program.

--9B095B5ADSN=_98C9A2A81C354B9EB49533476B1?WOM1.labinte
Content-Type: text/plain; charset=unicode-1-1-utf-7

This is an automatically generated Delivery Status Notification.

Delivery to the following recipients failed.

       user@hotmail.com




--9B095B5ADSN=_98C9A2A81C354B9EB49533476B1?WOM1.labinte
Content-Type: message/delivery-status

Reporting-MTA: dns;imc2-s4.hotmail.com
Received-From-MTA: dns;wim1-s4.hotmail.com
Arrival-Date: Thu, 19 Jun 2008 10:41:56 GMT

Final-Recipient: rfc822;user@hotmail.com
Action: failed
Status: 5.5.0
Diagnostic-Code: smtp;550 Requested action not taken: mailbox unavailable

--9B095B5ADSN=_98C9A2A81C354B9EB49533476B1?WOM1.labinte
Content-Type: message/rfc822

X-Message-Delivery: Vj0zLjQuMDt1cz0wO2w9MDthPTE=
X-Message-Status: n:0
X-SID-PRA: postmaster@mysyte.it <postmaster@mysyte.it>
X-Message-Info: JGTYoYF78jGyls5/72tozajnpJwPuqZsfwVQyN/kbtbDSBaIM9Z9PbGtyFD0pdXELKDRxAtOj1lG5Umu5so6+Q==
Received: from it11.xxxxxxxx.net ([xxx.x.xxx.xx]) by bay0-mc5-f6.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.2668);
	 Thu, 12 Jun 2008 14:10:20 -0700
Received: by it11.xxxxxxxx.net (Postfix, from userid 33)
	id 615B9489A1; Thu, 12 Jun 2008 23:10:19 +0200 (CEST)
To: user@hotmail.com
Subject: TomTom GO920T ad un prezzo speciale solo da Youbuy
From: "postmaster@mysyte.it" <postmaster@mysyte.it>
Message-ID: <20080612211019.5f0801817254@www.mysyte.it>
MIME-Version: 1.0
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: 8bit
X-Priority: 3
X-Mailer: vBulletin Mail via PHP
Date: Thu, 12 Jun 2008 23:10:19 +0200 (CEST)
Return-Path: postmaster@mysyte.it
X-OriginalArrivalTime: 12 Jun 2008 21:10:21.0275 (UTC) FILETIME=[B93792B0:01C8CCD0]

Best regards

[Milad]

This message didn't return to the bounce email (ex: bounce20080608103128.551e45224330@yourdomain.ext), it seems like it had been sent before you install the product.

Would you please copy and paste one of your sent messages (while the hack enabled) to see how they look?

[booktoweb]

thanks milad
i had make registration without activation by mail from 1 year
because i have a problem in yahoo mail ( it send to bulk mail )
so i will now ban yahoo mail from register
and return the activation to work
after this i will using your Product to manage all bad members that register with not valid mail

[Milad]

Don't be in a hurry and wait for the next beta.

[Sym0n]

Really interested in this, but I'm waiting for a final and preferably supported version.

Sy

[stwilson]

I still have not been able to get this to work. I have the bounced messages in my in-box. Here is the info from the To: field in the bounced message:

bounce20080616052150.d7e1a8553456@myohv.com

But the user's usergroup is never changed to the user group I identified in the MOD CP. Any suggestions?

[AngelBlue]

I've added some code so that bouncing users now get:
* Popup PM notification set to YES
* Receive admin emails set to NO
* Recieve email PM notification set to NO
* Thread subscription mode set to "no email" (if it was previously set to instant, daily or weekly email)
* All their thread and forum subscriptions set to 'no email'.
* A PM to explain the above.

To do this, I replaced the code in your "Move Users To Bounce Usergroup" plugin with this:

$bounces = $vbulletin->db->query_read("
	SELECT messageid, toemail
	FROM " . TABLE_PREFIX . "sent_email
	WHERE bounced = 1
");

if ($vbulletin->db->num_rows($bounces) > 0)
{
	$toemails = array();
	$messageids = array();
	
	while ($bounce = $vbulletin->db->fetch_array($bounces))
	{
		$toemails[] = "'" . $bounce['toemail'] . "'";
		$messageids[] = $bounce['messageid'];
	}
	$toemails = array_unique($toemails);

	// start added code here... define data for PM data manager

	$tonames = array();
	$touids = array();
	$myuid = 1;
	$fromuser = fetch_userinfo($myuid);
	$message = "We tried to deliver an email to you, but unfortunately, it bounced. To prevent us sending more mail to you while your email provider is not accepting it, we have deactivated all the email-related features of your account. To reactivate your account, go to your UserCP, and update your email address. You will then get a confirmation email. Clicking on the link in this confirmation email should re-enable your account, at which point you will need to visit your UserCP options page to re-enable the email notifications you wish to use.";

	$affected = $vbulletin->db->query_read("SELECT userid,username FROM " . TABLE_PREFIX . "user WHERE email IN (" . implode(', ', $toemails) . ")");
	while ($affected = $vbulletin->db->fetch_array($affected)) {
		$tonames[] = $affected['username'];
		$touids[] = $affected['userid'];
	}
	$tonames = array_unique($tonames);
	$pmdm =& datamanager_init('PM', $vbulletin, ERRTYPE_SILENT);
	$pmdm->set('fromuserid', '1');
	$pmdm->set('fromusername', 'EmailBot');
	$pmdm->set_info('reciept', false);
	$pmdm->set_info('savecopy', false);
	$pmdm->set('title', 'Email Broken!');
	$pmdm->set('message', $message);
	$pmdm->set_recipients(implode(";",$tonames), $fromuser['permissions']);
	$pmdm->set('dateline', TIMENOW);
	$pmdm->save();

	
	
	// this is stock mod code to move them to the different usergroup.
	$vbulletin->db->query_write("UPDATE " . TABLE_PREFIX . "user SET usergroupid = " .  $vbulletin->options['bm_usergroup'] . ", membergroupids = '', displaygroupid = 0 WHERE usergroupid NOT IN (" . $vbulletin->options['bm_excludeusergroups'] . ") AND email IN (" . implode(', ', $toemails) . ")");

	// and stock code to delete their message tracking info...
	$vbulletin->db->query_write("DELETE FROM " . TABLE_PREFIX . "sent_email WHERE messageid IN (" . implode(', ', $messageids) . ")");

	// continue custom code to update their user options...
	// stop them generating any more email!

	// pm popup = YES
	$vbulletin->db->query_write("UPDATE " . TABLE_PREFIX . "user SET pmpopup = 1 WHERE email IN (" . implode(', ', $toemails) . ")");
	
	// send notification email on PM - NO
	$vbulletin->db->query_write("UPDATE " . TABLE_PREFIX . "user SET options = (options - 4096) WHERE (options & 4096) AND email IN (" . implode(', ', $toemails) . ")");

	// admin email OFF
	$vbulletin->db->query_write("UPDATE " . TABLE_PREFIX . "user SET options = (options - 16) WHERE (options & 16) AND email IN (" . implode(', ', $toemails) . ")");

	// set autosubscribe = without email WHERE autosubscribe_with_email
	$vbulletin->db->query_write("UPDATE " . TABLE_PREFIX . "user SET autosubscribe = 1 WHERE (autosubscribe > 1) AND email IN (" . implode(', ', $toemails) . ")");

	// set emailupdates = 0 for their thread subscriptions
	$vbulletin->db->query_write("UPDATE " . TABLE_PREFIX . "subscribethread SET emailupdate = 0 WHERE (emailupdate > 0) AND userid IN (" . implode(', ', $touids) . ")");

	// set emailupdates = 0 for their forum subscriptions
	$vbulletin->db->query_write("UPDATE " . TABLE_PREFIX . "subscribeforum SET emailupdate = 0 WHERE (emailupdate > 0) AND userid IN (" . implode(', ', $touids) . ")");

	// finish
	unset($toemails, $messageids);
}

$vbulletin->db->free_result($bounces);

[Hornstar]

When I test my connection I got this message: 3 POP3 server greeting was not found

What does that mean?

Edit, changed pop host to an ip instead and that worked. Looking forward to the results of this. thanks.

Originally Posted by AngelBlue

I've added some code so that bouncing users now get:
* Popup PM notification set to YES
* Receive admin emails set to NO
* Recieve email PM notification set to NO
* Thread subscription mode set to "no email" (if it was previously set to instant, daily or weekly email)
* All their thread and forum subscriptions set to 'no email'.
* A PM to explain the above.

To do this, I replaced the code in your "Move Users To Bounce Usergroup" plugin with this:

$bounces = $vbulletin->db->query_read("
    SELECT messageid, toemail
    FROM " . TABLE_PREFIX . "sent_email
    WHERE bounced = 1
");

if ($vbulletin->db->num_rows($bounces) > 0)
{
    $toemails = array();
    $messageids = array();
    
    while ($bounce = $vbulletin->db->fetch_array($bounces))
    {
        $toemails[] = "'" . $bounce['toemail'] . "'";
        $messageids[] = $bounce['messageid'];
    }
    $toemails = array_unique($toemails);

    // start added code here... define data for PM data manager

    $tonames = array();
    $touids = array();
    $fromuser = fetch_userinfo(1);
    $message = "We tried to deliver an email to you, but unfortunately, it bounced. To prevent us sending more mail to you while your email provider is not accepting it, we have deactivated all the email-related features of your account. To reactivate your account, go to your UserCP, and update your email address. You will then get a confirmation email. Clicking on the link in this confirmation email should re-enable your account, at which point you will need to visit your UserCP options page to re-enable the email notifications you wish to use.";

    $affected = $vbulletin->db->query_read("SELECT userid,username FROM " . TABLE_PREFIX . "user WHERE email IN (" . implode(', ', $toemails) . ")");
    while ($affected = $vbulletin->db->fetch_array($affected)) {
        $tonames[] = $affected['username'];
        $touids[] = $affected['userid'];
    }
    $tonames = array_unique($tonames);
    $pmdm =& datamanager_init('PM', $vbulletin, ERRTYPE_SILENT);
    $pmdm->set('fromuserid', '1');
    $pmdm->set('fromusername', 'EmailBot');
    $pmdm->set_info('reciept', false);
    $pmdm->set_info('savecopy', false);
    $pmdm->set('title', 'Email Broken!');
    $pmdm->set('message', $message);
    $pmdm->set_recipients(implode(";",$tonames), $fromuser['permissions']);
    $pmdm->set('dateline', TIMENOW);
    $pmdm->save();

    
    
    // this is stock mod code to move them to the different usergroup.
    $vbulletin->db->query_write("UPDATE " . TABLE_PREFIX . "user SET usergroupid = " .  $vbulletin->options['bm_usergroup'] . ", membergroupids = '', displaygroupid = 0 WHERE usergroupid NOT IN (" . $vbulletin->options['bm_excludeusergroups'] . ") AND email IN (" . implode(', ', $toemails) . ")");

    // and stock code to delete their message tracking info...
    $vbulletin->db->query_write("DELETE FROM " . TABLE_PREFIX . "sent_email WHERE messageid IN (" . implode(', ', $messageids) . ")");

    // continue custom code to update their user options...
    // stop them generating any more email!

    // pm popup = YES
    $vbulletin->db->query_write("UPDATE " . TABLE_PREFIX . "user SET pmpopup = 1 WHERE email IN (" . implode(', ', $toemails) . ")");
    
    // send notification email on PM - NO
    $vbulletin->db->query_write("UPDATE " . TABLE_PREFIX . "user SET options = (options - 4096) WHERE (options & 4096) AND email IN (" . implode(', ', $toemails) . ")");

    // admin email OFF
    $vbulletin->db->query_write("UPDATE " . TABLE_PREFIX . "user SET options = (options - 16) WHERE (options & 16) AND email IN (" . implode(', ', $toemails) . ")");

    // set autosubscribe = without email WHERE autosubscribe_with_email
    $vbulletin->db->query_write("UPDATE " . TABLE_PREFIX . "user SET autosubscribe = 1 WHERE (autosubscribe > 1) AND email IN (" . implode(', ', $toemails) . ")");

    // set emailupdates = 0 for their thread subscriptions
    $vbulletin->db->query_write("UPDATE " . TABLE_PREFIX . "subscribethread SET emailupdate = 0 WHERE (emailupdate > 0) AND userid IN (" . implode(', ', $touids) . ")");

    // set emailupdates = 0 for their forum subscriptions
    $vbulletin->db->query_write("UPDATE " . TABLE_PREFIX . "subscribeforum SET emailupdate = 0 WHERE (emailupdate > 0) AND userid IN (" . implode(', ', $touids) . ")");

    // finish
    unset($toemails, $messageids);
}

$vbulletin->db->free_result($bounces);

Cool! Any chance of this getting packaged up. I would really love to get this installed on my board. It is now a requirement of AOL that you remove any permanently bounced emails from your mailing list or you risk getting blacklisted.