Email rejections due to DMARC
 

Some major mail servers, notably Gmail and Yahoo, have implemented a relatively new process/protocol/whatever called DMARC. This is causing emails sent by our users via our vBulletin, currently 3.8.7 PL 4, (i.e. email to another user, [s]send link to a friend, etc.[/s]) to be rejected.

vBulletin sends such messages with the sender's email address of record in the "From" field of the message. vBulletin does it this way so that the recipient can reply directly to the original sender's email address.

Best as I can make out by examining the online DMARC info, these messages are being rejected because the sending server (i.e. our board's server) does not match the server shown in the user's "From" address.

If this is correct, I *think* this could be corrected by having those user-initiated emails sent with our board's email address in the "From" field, and the sending user's email address in a "Reply-to" field.

Does anybody know how to accomplish this?

[Solution found. See: https://vborg.vbsupport.ru/showthrea...9#post2496459]

DMARC is only applied after both SPF and DKIM verifications fail.

All three of those are defined in the DNS entries for your site.

DKIM also requires both a public and private key on your server.

From Gmail about DMARC..

Thanks for the reply, but that's not been our experience.

Our normal mail (the stuff that has admin@ourbbsname.com in the From field) gets through just fine, including to Gmail and Yahoo.

It is just the messages that have our user's address in the From field that are getting rejected. When you go to the links provided in the rejection messages, they're pretty cryptic, but they seem to indicate (perhaps erroneously) DMARC failures.

Correct me if I'm wrong, but the fact that our "normal" mail is getting through okay seems to say that there is no problem with our SPF and DKIM settings (or at least one of them must be okay per what you said above), otherwise our normal mail would bounce too.

You can send a test mail to check-auth@verifier.port25.com and it will analyze your verifications and email you a report back.

I created a user with check-auth@verifier.port25.com as its email address and sent some messages to it.

Messages sent from my normal admin account, with a From address of admin@ourbbsname.com, produce the following summary (extracted from a much longer report):
In short, both SPF and DKIM checks passed.

And here's the summary for the same message sent from a user account, with a From address NOT located on our server, similar to the ones we are seeing rejected:
This still passes SPF and DKIM, but fails their Sender-ID check.

So again, it seems to point to a mismatch between From address and sending server being the culprit that is causing both Gmail and Yahoo to reject the messages.

I'm just looking for some guidance on how to get our own server-based address into the From field of all outgoing messages. And for those user-generated messages that would throw the user's own address into the From field, to put it into a "Reply-to" header instead.

Well, emails are sent by calling the "vbmail" function, and that function has parameters for specifying the From address as well as additional headers. So you should be able to do what you want by finding the right calls to vbmail() and changing the parameters. I would figure it out for you but I don't have the time to work it all out right now. I might be able to do it later if no one else has figured it out by then.

Edit: now that I think about it a little more, I'm guessing a plugin using hook mail_send could adjust the fields as necessary so that you wouldn't have to edit any files.

You need to add an "allow" or "include" to to your spf record for the domain like this:
a:vbulletin.org include:vbulletin.org
That allows the domain as a sender, i personally haven't had any issues with the email sending as i dont allow my users to use the email to a friend function or contact them by email but i'm guessing you'll have to add every domain that your users use.

Someone asked about the same thing here: https://vborg.vbsupport.ru/showthread.php?t=310799 and I didn't understand it, but after reading this thread I think he's having the same issue. If what he was told is true, then you should be able to fix it by using your own email in the "from" field, but put the user's email in reply-to.

No problem with sending OUR mail, with our address, via our server. Problem is sending mail originated by users with THEIR address in the From field. No way in hell I'm adding every domain used by over 50,000 registered users to our SPF record.

And btw, I misspoke earlier, and have struck out the erroneous text above. The email link to a friend option sends its message with the board's admin email address in the From field. This has caused me to have to add a bunch of boilerplate to those messages on the order of "DO NOT REPLY TO THIS MESSAGE as replies go to the BBS admins not to the person sending this message to you." Of course, nobody bothers to read the caveats...

--------------- Added 05 May 2014 at 18:16 ---------------

.. which is precisely what I'm asking about doing.

[Toddling off to the referenced thread to see what's there ....]

--------------- Added 05 May 2014 at 23:22 ---------------

FURTHER RESEARCH

The vbmail() function is defined in includes/functions.php, as follows:

function vbmail($toemail, $subject, $message, $notsubscription = false, $from = '', $uheaders = '', $username = '')

The parameters are:

* @param string Destination email address
* @param string Email message subject
* @param string Email message body
* @param boolean If true, do not use the mail queue and send immediately
* @param string Optional name/email to use in 'From' header
* @param string Additional headers
* @param string Username of person sending the email

The function is called in sendmessage.php in several different places, depending on the sort of email being sent. So it looks to me that I'll need to hack sendmessage.php for my user-generated message types to add in a specified From address that utilizes my mail server as the 5th parameter, and a "Reply-to" header as the 6th parameter.

Your whole problem might be solved as easily as adding this to your DNS entries...

spf2.0/pra a mx IP4:XXX.XXX.XX.XX -all

The only way to be sure is to try it.

This would be, I assume, the IP assigned to our server?

Would be worth a try, although the test results above don't indicate either SPF or DKIM failure.

--------------- Added [DATE]1399380674[/DATE] at [TIME]1399380674[/TIME] ---------------

As I read for further on this, I think you've got it backwards. I think this would say that people could send mail claiming to be "From" my domain via any SMTP server they pleased. This would make it easier for me to send mail using my address on the BBS via my home cable ISP's server, for instance, but it it would also permit any spammer to make up any address he wants on my BBS domain and send mail coming "From" there with impunity via any server he has access to.

I think what I would need is for the SPF record for each of my users' email domains to have an entry allowing their mail to be sent from my server, and there's no way that can happen.

Actually it means any mail coming FROM your IP address (with any email domain name in the from address) would pass. All other IP addresses claiming to be you would fail.

It's the simplest way to pass sender id verification.

I have done a LOT of researching on this. The problem is on the vbulletin side.
Look at this header from my site - it is the result of one member sending another a message. And yes, the actual emails have been changed.
Ok, now look at the FROM. It shows:
"ridenazi @ KTMTalk.com - The Absolute BEST KTM resource on the planet" <ktmXXX@yahoo.com>

It's [SAYS] that it's FROM ktmXXX@yahoo.com when in reality, it is from MY server. vBulletin does the "from" this way so that the "receiver" of the message can hit reply and have it go to the original "sender".

So THAT is the problem... the header says it is from ktmXXX@yahoo.com when it's actually from my server (ktmtalk.com).

To confirm, on your next "bounced email", go look at the headers, specifically the FROM. If the entire "from" shows ANYTHING except your mail server, it will fail DMARC.

btw: AOL and gmail have both started using this "standard".

-pat-

I have been having the exact same problem for about a month now.

We have the "secure email" option enabled (if not, this wouldn't be a problem).

Originally, it was ONLY when a member having a yahoo.com email address attempted to send an email to another user. It doesn't matter what the receiver's email address is...what makes it bounce back is when the SENDER'S email address is a yahoo.com email address.

More info on why can be found here:
http://marketingland.com/email-sende...emailmarketing

For about the past week or so, I have started to see aol.com email addresses (SENDER) get rejected for the same reason.

I too am hoping for a good solution to this...if nothing else, simply reject the email attempt at the "send" stage so the member knows immediately their email didn't go though (I already have a bold red large text notification in the email interface, but again...many users seem to overlook it (somehow) and submit their email anyway...

it's going to start happening more and more.... with more an more providers (aol, gmail, etc).

if the SENDER (ie, the FROM) contains ANYTHING except YOUR SERVER, it will fail DMARC.

I'll post a solution in a bit - I want to confirm a few other things first.

We also have this issue on my vB3 forum, Ive been trying to figure out whats going on for 3 weeks (on and off) all our e-mails to Gmail are just getting bounced.

Thanks, Pat, I've found the same. It may not exactly be DMARC causing the rejection, but an additional Sender-ID check being run by more and more servers -- not sure if this is part of DMARC or something separate, but no matter, it causes rejections either way.

I'm testing a hack of sendmessage.php right now that sends these user-generated messages with the default bbs address in the "From" field, and with the sending user's email address in an additional "Reply-to" header. This way the message should pass the Sender-ID test (the From address is on the sending server) but the recipient can still reply to the actual sender thanks to the Reply-to header.

Test messages I've sent via my test board to one of my other email addresses show the headers to be getting sent the way I want. Don't know yet if this really solves the problem when sending to gmail, yahoo, etc.

Paul... vb3, vb4, vb8000... it will happen on any vb version that puts anything other than your server as the from (ie, the sender).

The next time one bounces, go look at the header... you'll see the FROM issue.

--------------- Added [DATE]1399391922[/DATE] at [TIME]1399391922[/TIME] ---------------

That should do the trick... it puts the REAL "from" (your server) in the "from" (ie, the "sender") and your "reply to" make a reply go to the correct email address (and not the forum's admin).

I need to go take a qwik check and find all occurances where VB changes this "from" and then I'll be able to post a "one size fits all" solution.

--------------- Added [DATE]1399392384[/DATE] at [TIME]1399392384[/TIME] ---------------

Click this link - http://www.dmarc.org/faq.html#g_13
It's a frigging dissertation on dmarc... just scroll down 40 or 50 pages to the section titled "Why are messages I send on behalf of visitors to my website being blocked?"

For those not wanting to read thru all that crap, it says:
Why are messages I send on behalf of visitors to my website being blocked?

Here's my hack to sendmessage.php. This is for vB3.8.7 - don't know if there have been syntax changes in later versions. The keys here are the 5th and 6th parameters passed to vbmail().

5th = the From email address. If not specified, or specified as a null (''), the default address for the bbs will be used.

6th = any additional headers you'd like to define. This is where we'll define the Reply-to header.

In my vB 3.8.7 vbmail() is called three times in sendmessage.php.

1. First time is for ContactUs mails. I don't currently have any trouble with these, probably because MY server isn't making DMARC/Sender-ID checks on incoming mail. Or maybe it's just that, even though it has the sender's email shown as From, in reality it is going from and to the same server (??).
   
   Find:
   PHP Code:

   ---

   vbmail($destemail, $subject, $message, false, $vbulletin->GPC['email'], '', $name); 


   ---

   Replace with:
   PHP Code:

   ---

   vbmail($destemail, $subject, $message, false, '', "Reply-To: {$vbulletin->GPC['email']}\n", $name); 


   ---

2. Second is for Send-A-Link-to-Friends mails. These now go thru okay because they show as coming from my admin email address. But would be MUCH better if they also included a Reply-to header so the recipient could reply to the real sender directly, instead of replies coming back to me.
   
   Find:
   PHP Code:

   ---

   vbmail($vbulletin->GPC['sendtoemail'], $vbulletin->GPC['emailsubject'], $message); 


   ---

   Replace with:
   PHP Code:

   ---

   vbmail($vbulletin->GPC['sendtoemail'], $vbulletin->GPC['emailsubject'], $message,'','',"Reply-To: {$vbulletin->userinfo['email']}\n"); 


   ---

3. Third is for Email-to-another-bbs-user mails. These are the ones currently giving me fits.
   
   Find:
   PHP Code:

   ---

   vbmail($userinfo['email'], fetch_censored_text($vbulletin->GPC['emailsubject']), $message , false, $vbulletin->userinfo['email'], '', $vbulletin->userinfo['username']); 


   ---

   Replace with:
   PHP Code:

   ---

   vbmail($userinfo['email'], fetch_censored_text($vbulletin->GPC['emailsubject']), $message , false, '', "Reply-To: {$vbulletin->userinfo['email']}\n", $vbulletin->userinfo['username']); 


   ---

PERFECT! Thanks!
I tested both yahoo and aol email addresses (SENDER) and both went through fine now.
Replying to the resulting email does indeed direct it to the real sender.

By the way, my installation is 4.2.2 and there was no change in syntax on this hack.

Thanks for the feedback. It seems to be doing what it's supposed to here, too.

That's good to hear for when (if?) I finally get around to updating this beast.

I really hate to do this by hacking a php file, but I couldn't figure out how to do this via a plugin.

That wasnt the issue, as I said, it was rejecting all e-mails, the vast majority of them being subscription e-mails, which come from the sender domain.

Basically gmail has become very picky/sensitive to the slightest error. I went through our whole email set up and found a couple issues, one of which I suspect was probably the culprit, but Im not going to reverse them to confirm. Basically our IPv6 reverse dns was incorrect, but I dont think that was the problem, more likely was that the domain in our MX record (mail.mydomain.com) was pointed to a CNAME record (it CNAMED to mailserver.mydomain.com). Apparently this is not RFC compliant, so I changed them to use the direct server name which is defined as an A record.

I've found an error in the replacement ContactUs code (point #1 above).

The original code, which used $vbulletin->userinfo['email'] to retrieve the poster's email address for insertion into the Reply-to header, only works for logged-in registered users of the board. If you allow un-logged-in visitors to use the Contact Us form, the email address they enter will not be picked up for the Reply-to header.

Instead, use $vbulletin->GPC['email'], as in the original vB coding. This works both for logged-in users and for visitors.

I've edited the code in the original post.

Applied all mentioned here but for some reason every once in a while i get this.

This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:

kalembo@hotmail.com
SMTP error from remote mail server after end of data:
host mx2.hotmail.com [207.46.8.199]: 550 5.7.0 (BAY004-MC6F11) Message could not be delivered. Please ensure the message is RFC 5322 compliant.

any info?

many thanks I always effected with hotmail blocking ???? ???? ????

Thanks makaiguy, resolved the Yahoo policy rejects.

And now there's this...

This message has been rejected due to content judged to be spam by the internet community\302\240IB212

Read this sweet information: http://x.co/crbounce

552 This message has been rejected due to content judged to be spam by the Internet community. IB212

The email message contains a link, attachment, or pattern caught by our filters as spam. Please include an option to opt out in your email messages. Then check your sending lists to ensure you are only sending to recipients who have selected to opt in to receiving your mail. If you feel this message has been flagged as spam erroneously, please be sure to obtain a copy of the original message attempting to be sent, and then contact support.

OH, but OPTING OUT is a LINK isn't it?

Contact their support? Their email is do-not-reply.

Have run into the same thing here, with the same frustration. They're occurring for me with forum and thread subscription email notices. I guess their software is too dumb to recognize that "unsubscribe" links are the same thing as "opt out" links.

I've just edited the boilerplate phrases that generate these messages so that everywhere they said "unsubscribe" they now say "unsubscribe (opt out)". Too soon to know if this makes any difference.

It's been almost two months now and I haven't received any more of those rejection notices, so maybe this worked.

It is not so much having it in your email but on your site when people register. When you do the request with Google and Microsoft to stop blocking your email you have to submit the opt in/out link along with other stuff.

What gets us hammered as well is if someone marks you as spam.

I went through email a couple months ago and had to do my dns settings over completely including new name servers. it is vital that your sending ip address matches your domain.

At any rate, glad it is working out for you now.

It really ticked me off when this happened to me because we do not spam anyone and were being blocked only because our email server settings were not up to standard.

Would be great if we could get 1 and 3 added to the core in vB 4.2.3 Beta 4

I had to change

to

to make it RFC compliant and get it working with some e-mail provider.

Changes are: the capital T in Reply-To: and added space after Reply-To:

Thanks. That might explain the RFC Compliance problem mentioned in post 22.

I've edited the original post with these changes.

Thanks guys, just to confirm that this is working well for the Contact Us form in 4.2.3 PL2 :up::up::up:

This also works great on 3.8.11. I'm surprised this change wasn't in this newest (and maybe last) version of VB3.

No one logged it in Jira.

Had this exact issue when pushing emails through Office 365. This hack gets it working in vb5 5.4.5, but does not have the Reply-to because I didn't want to mess with figuring that out. So it sends and receives as the same user that you define, but the message body still has the user's email address .

In this file: ./core/vb/api/contactus.php

Replace:

With:

Cleaned up the spf/dkim/dmarc behavior of my forums recently, and this same issue cropped up with the contact us links. The fix posted in this thread still works fine - and the emails are now coming through just fine once again.