Major Additions - Email Integration (New Threads/Replies by and through email!)

This modification allows you to mimic email lists such as yahoo groups through your forums!

Complete with the user's receiving emails for new threads and replies AND being able to reply directly from email or sending a new email to start a new thread!

Just like email lists, you will need a separate email box for EACH forum that you have this enabled for!!

After installing this modification in the forum manager you will enter the settings for each forum to connect to it's own email box.

After doing so, members can subscribe to each forum that this mod is enabled for to receive notifications for new threads and replies.

Any posts in that forum, will be sent to them via email in one of two formats that you as the administrator decide. One format has a "reply box" and is the most sure fire way of not getting excess email crap ont he forums. The second format (noted as "Old" format) keeps an email in its base pure form with some attempts at quoting messages. Examples of both are attached below.

The users can then reply to that email (which will then be processed and posted to the forums through the cron job that runs every 10 minutes), or they can send a new email to the email address and a brand new thread will be created!

Essentially mimicing an email list! Suddenly those users that hate forums, can still be dragged into the conversations and help keep the forum's activity level up!

IMAP
You MUST have php imap functions enabled on your server to use this modification! You do not need to have IMAP installed, just have to enable php to use the functions by recompiling it with imap enabled!

If you are upgrading from the version
for vbulletin 3.6.x , see below!

BUGS!
These are the known bugs to date:

• Unconfirmed bug that email processing does not handle Japanese characters.

Unsupported Items
These are the items that are not currently supported:

• Some custom bbcode modifications will not display like they do on the forums in the html emails. Like glow, and the table mod.

TO UPGRADE

• Please see the file in the zip for upgrading!! If your upgrading from version 2.3.X you will need to reverse some file edits that are no longer nessecary!

NEW INSTALL

• See instructions in the zip file.

VERSION HISTORY!

• 2.5 Release
• • New Features
    • Email Integration now uses vbmail! This DOES require one small file edit.
• 2.5.1 Release -
  • New Feature -
    • Added some debug triggers in the cron file. Turning on debug will stop email sfrom being deleted after they are processed and if a message errors out, the error email will be sent to an admin specified email address which is set in the cron file.
  • Bug Fixes -
    • Fixed incorrect call to plain text parsing. (this is the cause of the duplicate posting/undefined function errors)
    • Corrected security hole.
    • Corrected a typo in the error message for attachments that are too large.
    • Rebuilt the method of determining attachment types vs max size to handle more then 2 attachments. Previous code would not process more then 2 attachments.
    • Added a call to determine the parentid for replies if one isn't specified prior to the post being added to the database from emails.
    • Fixed a bug where previous messages were blank in new layout for posts that didn't have a parentid specified.
• 2.5.2 Release
  • Bug Fixes
    • Fixed bug with new users not being automatically subscribed to forums when it is enabled for their usergroup.
    • Added a dirty regex fix for utf-8 encoded filenames.
• 2.5.3 Release
  • Bug Fixes
    • Fixed bug that displayed "Set All Instant" on User CP Subscribe Forums page when no forums were setup for Instant Email Notification.
    • Fixed bug that allowed forums that were set inactive to appear on the User CP SUbscribe Forums page.
    • Fixed bug with the how the "New Layout" processs improving how it tries to remove the first layer of > signs in replies.
  • Improvement/Bug Fix
    • Changed the field in the Forum Manager for forum['ei_usergroups'] so that it now controls not only who can reply via email. But who can receive instant notification emails. It is now titled "Please enter the usergroup IDs that can use this feature."
    • Added the phrase "You may increase the space between the above lines as needed." to the "New Layout" email templates to alleviate user confusion.
• 2.5.4 Release
  • Bug Fixes
    • Added a quickfix for a bug that caused a function error when replying to a thread that a user was subscribed to while in a forum that was enabled for Email Integration.
  • Error Checks Added
    • Added a function exists check to the cron file and the search settings page to warn people when they have not correctly enabled imap functions.
• 2.5.5 Release
  • Bug Fixes
    • Added a check to subscribeforum.php to not display subscription options for forums that are links.
    • Fixed a bug that caused the cron to incorrectly list the username and signature in the emails. (The post processed to the forums were correct, it was just the emails that were screwy sometimes.)
    • Slightly modified phrase and variables for replies so that if a post has a title, it'll use that title for the email, if not, it'll use the thread title like before.
• 2.6 Release
  • Bug Fixes
    • Joining an unmoderated public group will now auto subscribe you per the usergroup settings.
    • NO CONFLICTS with Instant Thread Subscription! See details in below post.
  • New Stuff
    • User Option Allow Auto Subscribe - This is an admin allowable, user option that let's the user choose to not be auto subscribed.
    • User Option not receive own posts - A new user option that lets them select to NOT receive their own posts via email.
    • Editted the phrase for the Auto Subscription script to tell people what to do if there are no listings.
    • Added error checks when saving a usergroup for if auto subscriptioin is entered with a forumid for a forum that does not have email integration enabled and also checks that the usergroup you are saing is entered in the forum manager as allowed to use email integration.
• 2.6.1 Release (BETA VERSION)
  • Hot Fix
    • It appears I was unseting a variable too soon. I've corrected this and it has resolved the issue with multiple posted replies on my test server.

Most of these will be slow to make it in til I know I've got the code to handle just about every type of email we could receive But in anycase, here are the ideas for future versions!

• End User Option to subscribe to just receive new thread emails, or receive all new thread & reply emails (current version does all new threads & replies).

Thanks to RedTyger, Bob Denny, Ed Kohwley and Chris McKeever for their additions and assistance with portions of this code.

Huge thanks also goes to the many individuals that have helped test various incarnations of this mod.

MAD PROPS TO: cgmckeever for a great quoting regex for the old format, and all the mods at www.4winmobile.com for helping test version 2.4 prior to release!!

You guys rock!

Please feel free to donate to my continued work on this modification!! It let's me spend more time on it!
Make a Donation!

Version 2.6 will work with both 3.6.8 and 3.7x

Upgrading from version for 3.6.x

• Due to using different product ID's due to the changes that will be coming up for the version for vbulletin 3.7.x you will need to uninstall the 3.6.x version and install the 3.7.x version in this thread. My apologies for the inconvienance, I've been unable to determine an alternative with the 3.6.x version still in development due to the last few bugs.

The #.X versions are the solid tested versions.

The #.X.X versions are the beta versions. New features and bug fixes will be worked in there and then when I have a solid version it'll be released as the next #.X

Thank you!!

~ Cyricx

[Cyricx]

Well to kinda give an idea of how the mail queue system runs from my experience dabbling in the bugger. Perhaps someone can correct me if I'm wrong.

When you post a reply with this modification, it'll run some queries to determine who all should receive an email. Then it will check to see if you have it set to run in batches. If you do have it set to run in batches, it'll add those emails to the mail queue table.

The next time a user refreshes any page on your site it'll run check how many you want to send per batch, grab those emails from the database, send them and delete those rows.

The next time it is refreshed it'll repeat this process.

I'm not one hundred percent certain on how long it waits between batches to allow another batch to be sent but I believe it's only about 5-10 seconds.

I have sent as many as 100 emails at one time with this mod prior to the vbmail days and experienced no adverse server side effects. Though I run on a VPS server. Others didn't hit problems until they tried to send 1000 emails at once! Hehe. So If you have problems with your mail backing up, try setting your batches higher, to like 10 or 20 and then monitor the system.

The reason I like the new layout is it's much more aesthetically pleasing for the forums and with the "reply box" it kinda gently encourages the users to visit the forum rather then just slink along on emails.

Which in the end.. is what we want... them visiting our forums. Going to the other sections of the site and heck.. in some cases... clicking on our ads

[snoopy5]

Originally Posted by Cyricx

...

The next time a user refreshes any page on your site it'll run check how many you want to send per batch, grab those emails from the database, send them and delete those rows.

The next time it is refreshed it'll repeat this process.
....

Hi

this would be a problem for me. I need this e-mail notification sent out without a user visiting the page to refresh it.

I have a few boards, that have not enough traffic. So maybe one user every 2 days finds the forum. Sending just the e-mail out, after 2 days of the original posting would be too late.

I know of a different forum-software (www.discusware.com), that has this e-mail reply functionality too. And they use also cronjobs for it. But I can schedule the cronjob, that because of the cronjob every 3 minutes it checks for new messages (no matter whether somebody hit the forum or not) and sends the e-mail out.

A. Could you implement this too?

B.

I need always to send out e-mails for around 500-1000 users with this for each vb-borad I have on the same server. I have several vb-boards on the same server. And each board needs to send those e-mail to this number of people for each vb-board.

Would be the mailserver/serverload because of this to be too high? I use a dedicated server.

Thanks in advance

[Cyricx]

Originally Posted by snoopy5

Hi

this would be a problem for me. I need this e-mail notification sent out without a user visiting the page to refresh it.

I have a few boards, that have not enough traffic. So maybe one user every 2 days finds the forum. Sending just the e-mail out, after 2 days of the original posting would be too late.

I know of a different forum-software (www.discusware.com), that has this e-mail reply functionality too. And they use also cronjobs for it. But I can schedule the cronjob, that because of the cronjob every 3 minutes it checks for new messages (no matter whether somebody hit the forum or not) and sends the e-mail out.

A. Could you implement this too?

B.

I need always to send out e-mails for around 500-1000 users with this for each vb-borad I have on the same server. I have several vb-boards on the same server. And each board needs to send those e-mail to this number of people for each vb-board.

Would be the mailserver/serverload because of this to be too high? I use a dedicated server.

Thanks in advance

I do have a beta script that could be potentially run from the local server on a cron job. My expertise on setting up cron jobs for a server is nil though. If you have knowledge about those I'll send you the file I have that can be run from any directory. Just PM me your email address.

As for the mailserver/serverload. I can't say for sure, depends on your server settings. I believe the individual that experienced an issue when sending 1000 emails was on a VPS, so I would venture to say that on a dedicated server you will likely have no issues.

Either way the script would not fail, only the server could. If you set up vbmail to run in batches of a few hundred though, I think you would definately be rockin strong.

[Cyricx]

UPDATED!!

Here's the update message I sent out -

As many have noticed I'm on a bug squashing and obliterating mission!

I had hoped to get a few more minor bugs fixed in this version but
finally found the most elusive and meanest one which was causing people to
get errors when replying to some threads.

The criteria to cause this bug was:

A user must be subscribed to the thread, the thread must be in a forum
that email integration has been enabled. That user must not be
subscribed for instant notification to the forum... only the thread.

Then you would get the error.

I've added a quick fix for the error in 2.5.4

Please upgrade.

If you notice any quirks... bugs... or issues... PLEASE let me know.
I'm devoted to making this a solid modification but I rely on your input


Thank you for your support and patience!

~ Chris

[Cyricx]

Just wanted to give everyone a heads up that I hadn't dropped off the planet and I'm working on the bugs that have been reported.

Here is a list of known confirmed bugs.

These are ones that I have NOT been able to duplicate so may actual be due to issues with another modification:

• Duplicate replies when posting in a forum that EI is turned on for.
• Headers when sent through a forwarder are lost when Gmail receives them. - Not a bug! Server settings required setting the delimiter in EI options to \n
• Occasionally lists a newthread poster as "unregistered" in the email that is sent out. - FIXED in 2.5.5

These are ones that I HAVE been able to duplicate in my test board and I'm working on fixes for them now:

• Does not correctly decode iso-2022-jp emails.
• Does not carry reply title to email. - FIXED in 2.5.5
• Subscribeforums.php shows forums that are links. - FIXED in 2.5.5

Please let me know if you are also experiencing any of the unduplicated bugs so that perhaps I can compare your site to the one having the issue to determine what may be the joining factor

The few bugs we are down to seem to only affect one site out of 40 LOL. So the criteria for bugs is getting very obscure... but hey... like I said.. I'm committed to making the most solid major addition modification on vb.org

[beishe8]

Originally Posted by Cyricx

...I'm committed to making the most solid major addition modification on vb.org

Good on you Cyricx, keep going.
vB just cannot realise how important it is.

Thanks for your hard work!

[Cyricx]

Thank you!

I'm going to jump straight to 2.6

Still waiting on some feedback on two obscure "bugs" so starting the coding for the other stuff that I want to get in asap.

So far coded is the user option to not receive emails for your own posts
Also, admins can allow users to choose if they do NOT want to be autosubscribed.

Those are now already coded and in.

Next I'm going to check out some stuff with how the delimiter is built to see if I can tighten that up. Work on the error message for the usergroup manager so that it will give you an error if you try to autosubscribe a group that is not setup to be allowed to use the feature in the forum manager and lastly... friendly involvement with the thread subscription

I'm a busy bee

[Cyricx]

Well... I went from 2.5.4 and went straight to a 2.6 release

Prolly had something to do with spending 8 hours doing nothing but working on this code hehe

2.5.5 was created in the interim and not released as while working on some bugs I made some huge breakthroughs in what I wanted done for 2.6... which is the coexistance with thread subscriptions as well as the user options for not getting their own posts and (if the admin allows them to) users can select to not be autosubscribed.

Check my third post in this thread for details on how the coexistance with thread subscriptions work.

I'm undecided at this time if I'm going to try to work thread subscriptions to let yu reply via email or not. It could require ALOT of file editting in the functions_newpost.php file which I'm trying to avoid as many file edits as possible.

All in all, I'm excited to feel like this may be my final version for awhile with the exception of working on the last confirmed bug of it not handling japanese characters


Several have asked for the cron jobs not to depend upon vbulletin traffic to work.

I'm really not up for writing something that intensive... it would involve creating a queue system as well as trying to teach people how to create cron jobs on their server... yeah.. not my piece of the pie

Plus I'm very happy with the way that this works with vbulletin. My forums typically have 1-2 visitors on them at a time right now and it's still working great.

The point of this is NOT to replace vbulletin. I want to encourage the users to come to the forums and hopefully eventually weed them off the mailing list lifestyle and into the forum lifestyle so to speak

In anycase, I do have a script that could run by cron job, but it still uses vbmail which requires traffic to activate vbmail and send the emails.. so.. hey.

You could modify it to use regular php mail but I would prolly advise against it as it could crash a server if you tried to send off 1000 emails at once!

Enough babble... hope you enjoy the latest version and donations are certainly appreciated

Thank you!

[popmyzit]

Hello Cyricx,

I have been having a problem. Using the gmail settings you posted i get this when running from the tasks:

Warning: imap_open() [function.imap-open]: Couldn't open stream {pop.gmail.com:993/pop3/ssl}INBOX in /includes/cron/emailintegration.php on line 89

Failed to connect to the mailbox for forum - xxxxxx forum

also happens when i try selfsigned ssl pop3:

Warning: imap_open() [function.imap-open]: Couldn't open stream {pop.gmail.com:995/pop3/ssl/novalidate-cert}INBOX in /includes/cron/emailintegration.php on line 89

Failed to connect to the mailbox for forum - xxxx forum

--------------------
Now whats weird is i have POP3 on the forum settings not Imap...

Any idea what is going wrong? All the settings are right (username, pass, etc)

[Cyricx]

Make sure you have enabled pop3 in your gmail box.

Per the third post in this thread

• Can I use a Gmail account to pull the emails from?

You most certainly can.

Login to your mailbox at gmail, go to Settings -> Forwarding and POP/IMAP

Enable POP for the account, either from here on, or for all mail. Be sure to select archive or delete for what to do after the messages are downloaded. Otherwise the system will continue to repost the emails!!

In your forummanager use the folowing settings:

email address to reply to: username@gmail.com
hostname: pop.gmail.com
username: username@gmail.com
password: password
Type of connection: select "POP3 with SSL (Self signed certificate)"

Done!

If you don't enable your gmail box to allow people to connect to it via pop3 or imap it will give that error