Email Integration (New Threads/Replies by and through email!)
 

• Unconfirmed bug that email processing does not handle Japanese characters.

Unsupported Items
These are the items that are not currently supported:

• Some custom bbcode modifications will not display like they do on the forums in the html emails. Like glow, and the table mod.

TO UPGRADE

• Please see the file in the zip for upgrading!! If your upgrading from version 2.3.X you will need to reverse some file edits that are no longer nessecary!

NEW INSTALL

• See instructions in the zip file.

VERSION HISTORY!

• 2.5 Release
• • New Features
    • Email Integration now uses vbmail! This DOES require one small file edit.
• 2.5.1 Release -
  • New Feature -
    • Added some debug triggers in the cron file. Turning on debug will stop email sfrom being deleted after they are processed and if a message errors out, the error email will be sent to an admin specified email address which is set in the cron file.
  • Bug Fixes -
    • Fixed incorrect call to plain text parsing. (this is the cause of the duplicate posting/undefined function errors)
    • Corrected security hole.
    • Corrected a typo in the error message for attachments that are too large.
    • Rebuilt the method of determining attachment types vs max size to handle more then 2 attachments. Previous code would not process more then 2 attachments.
    • Added a call to determine the parentid for replies if one isn't specified prior to the post being added to the database from emails.
    • Fixed a bug where previous messages were blank in new layout for posts that didn't have a parentid specified.
• 2.5.2 Release
  • Bug Fixes
    • Fixed bug with new users not being automatically subscribed to forums when it is enabled for their usergroup.
    • Added a dirty regex fix for utf-8 encoded filenames.
• 2.5.3 Release
  • Bug Fixes
    • Fixed bug that displayed "Set All Instant" on User CP Subscribe Forums page when no forums were setup for Instant Email Notification.
    • Fixed bug that allowed forums that were set inactive to appear on the User CP SUbscribe Forums page.
    • Fixed bug with the how the "New Layout" processs improving how it tries to remove the first layer of > signs in replies.
  • Improvement/Bug Fix
    • Changed the field in the Forum Manager for forum['ei_usergroups'] so that it now controls not only who can reply via email. But who can receive instant notification emails. It is now titled "Please enter the usergroup IDs that can use this feature."
    • Added the phrase "You may increase the space between the above lines as needed." to the "New Layout" email templates to alleviate user confusion.
• 2.5.4 Release
  • Bug Fixes
    • Added a quickfix for a bug that caused a function error when replying to a thread that a user was subscribed to while in a forum that was enabled for Email Integration.
  • Error Checks Added
    • Added a function exists check to the cron file and the search settings page to warn people when they have not correctly enabled imap functions.
• 2.5.5 Release
  • Bug Fixes
    • Added a check to subscribeforum.php to not display subscription options for forums that are links.
    • Fixed a bug that caused the cron to incorrectly list the username and signature in the emails. (The post processed to the forums were correct, it was just the emails that were screwy sometimes.)
    • Slightly modified phrase and variables for replies so that if a post has a title, it'll use that title for the email, if not, it'll use the thread title like before.
• 2.6 Release
  • Bug Fixes
    • Joining an unmoderated public group will now auto subscribe you per the usergroup settings.
    • NO CONFLICTS with Instant Thread Subscription! See details in below post.
  • New Stuff
    • User Option Allow Auto Subscribe - This is an admin allowable, user option that let's the user choose to not be auto subscribed.
    • User Option not receive own posts - A new user option that lets them select to NOT receive their own posts via email.
    • Editted the phrase for the Auto Subscription script to tell people what to do if there are no listings.
    • Added error checks when saving a usergroup for if auto subscriptioin is entered with a forumid for a forum that does not have email integration enabled and also checks that the usergroup you are saing is entered in the forum manager as allowed to use email integration.
• 2.6.1 Release (BETA VERSION)
  • Hot Fix
    • It appears I was unseting a variable too soon. I've corrected this and it has resolved the issue with multiple posted replies on my test server.

Most of these will be slow to make it in til I know I've got the code to handle just about every type of email we could receive :) But in anycase, here are the ideas for future versions!

• End User Option to subscribe to just receive new thread emails, or receive all new thread & reply emails (current version does all new threads & replies).

Thanks to RedTyger, Bob Denny, Ed Kohwley and Chris McKeever for their additions and assistance with portions of this code.

Huge thanks also goes to the many individuals that have helped test various incarnations of this mod.

MAD PROPS TO: cgmckeever for a great quoting regex for the old format, and all the mods at www.4winmobile.com for helping test version 2.4 prior to release!!

You guys rock!

Please feel free to donate to my continued work on this modification!! It let's me spend more time on it! :)
Make a Donation!




Upgrading from version for 3.6.x

• Due to using different product ID's due to the changes that will be coming up for the version for vbulletin 3.7.x you will need to uninstall the 3.6.x version and install the 3.7.x version in this thread. My apologies for the inconvienance, I've been unable to determine an alternative with the 3.6.x version still in development due to the last few bugs. :)

The #.X versions are the solid tested versions.

The #.X.X versions are the beta versions. New features and bug fixes will be worked in there and then when I have a solid version it'll be released as the next #.X :)

Thank you!!

~ Cyricx

Where are the files?

Beat me to the punch! :)

They are uploaded now!



FAQ

• How do the emails work?

• Error - Undefined Function

• Error - Couldn't open stream

• Can I use a Gmail account to pull the emails from?

• Smilies don't show up in html emails!

How does it work with Thread Subscriptions?

Supper mod, Nominated

Fatal error: Call to a member function query_first_slave() on a non-object in C:\xampp\htdocs\global.php on line 247

Any clues as to this error? I get that when I run the cronjob.

Cheers

Edit:

I suspect the issue is with the file edit. Please be sure you have followed the file edit needed in the /includes/class_mail.php file.

Yup, the edit is there and just the way it should be......

I will try it!

Cybertims,

I've removed the references to _slave in the queries in some of the plugins and the /includes/cron/emailintegration.php file.

Please reimport and overwrite with the xml included in this zip and overwrite the /includes/cron/emailintegration.php file.

If that doesn't resolve your issue let me know and we will take a closer look at things :)

Interesting, but even though that error happens when I manually run the cron, the system still works and everything posts ok on the Forum and emails go out ok.

Will try this new xml file as well.....

Cheers

Right on, remember to upload the new cron file too.

My 2.4 code made some query references to the slave database. I suspect that you have different settings then those running on 3.6 that have upgraded so far :)

Found the error in the cron file :)

New zip attached.

Mah bad... trying to help someone figure out a way to setup a system cron to run the file instead of waiting for vbulletin :)

Hey Cyricx,
just installed. thanks for the great work!!

Got error going to maintenance email area with mysql error:
Database error in vBulletin 3.7.0 Beta 4:

Please go to your Usergroup Manager and be sure that you have entered in a forumid for the usergroups that you wish to enable auto subscriptions for.


In version 2.4 you could actually turn on auto subscription for a usergroup and not enter in the forumids. This would spiral to a few other errors such as the one you mentioned above :)

2.5 will not allow you to not enter in a forumid if you are turning on auto subscription. So I'm fairly certain that when you installed 2.4 that mistake was made. :)

I've put in a safety net to ensure that fumble isn't made with these versions going forward :)

Hello Cyricx,
also when registering I still get:


Database Error Database error
The Forum database has encountered a problem.
Please try the following:

* Load the page again by clicking the Refresh button in your web browser.
* Open the www.mysite.com home page, then try to open another page.
* Click the Back button to try another link.

The www.mysite.com forum technical staff have been notified of the error, though you may contact them if the problem persists.

We apologise for any inconvenience.

maybe i will try i reinstall, i was upgrading from a previous version... that may be the problem.

Update: Yes Updating was the problem. Make sure you uninstall, then reinstall fresh clean install. worked for me.

The most common problem people have from upgrading from the older versions like 2.3 is forgetting to reverse the file edits or to put in the new file edit in the class_mail.php file.

Please follow the instructions in the UPGRADE.txt file closely :)

Let me know if you have any other issues Popmyzit and I'll work to resolve them asap!

Everything is fine after installing everything fresh. I will be looking for any bugs when people start sending emails

thanks again!

Hi

I am very interested in this hack. I only need a slight modification. I would be willing to pay for this.

I need this mailinglist functionality depending on the usergroup. I do not want that every user can use this feature, only premium members and Moderators shall be allowed to use this feature.

Would that be possible?

Thanks in advance

P.S.: Where is the difference between this hack above and this one:

https://vborg.vbsupport.ru/showthread.php?t=151222

Isn't the potential for this to backup the mail server pretty high? This could essentially generate a lot of emails if the site is active. If i am incorrect, I apologize. I am just thinking of the mass email that this could generate in addition to the email the site already sends.

very good question...

Can anybody answer this?

This is very possible and I'll plan it in for the next version. Right now it allows you to restrict who can reply via email, but ti doesn't restrict who can subscribe to instant email notifications.

Let me check out some of these bugs and get them cleaned up and I'll code that in for the next version as well :)


In the older versions this was a large issue. However, now the modification uses the vbmail function which allows you to define how many emails it can send at once. If it needs to send more then that number then it stores them in the database and periodically processes as vbulletin normally does :)

Ｈｅｌｌｏ！

Ｔｈａｎk you very much for making nice huck.
I installed this to 3.7 and seems working well.

My site is Japanese and using UTF-8 code.

When I make new thread, the mail for informing the content of new thread
can be sent without trouble.

When I reply by using the mail format, then
VB can receive the mail and show the conten, but can't read as Japanese.

Japanese mailer use ISO-2022-JP when sending mail.
I am sure so this is the reason that VB can't show Japanese when receive the mail.

Please advice where I should fix to solve this.

I'll need to check into this and ensure that the mail decoder I use is setup to read that format.

I'll get back to you soon. Gonna be digging on this product hard today to fix a few of the reported bugs.

Atsuko, I sent you a PM with an email address. If you could please send me an email encoded as you mention to the email address in the private message, I'll work on the script to resolve your issue :)

Thank you!

But if you process "x" amount of emails per hour, and the amount generated exceeds that number, you would still get a backup. It would all be qued to be delivered, but the mail que could be potentially backed up by days.

I am not just thinking of the mail generated by this script. i am trying to encompass all of the mail that the site already sends. With this added load, on active sites, couldn't this become an issue?

I think this is great for less active communities, but fear it would slow delivery to even normal email confirmations if the site is extremely active.

I think you underestimate the amount of time it takes vbulletin to process and deliver those emails :)

While I definately agree that putting it on a forum where you could have 1000 members autosubscribed to it would add a bit of a delay in the posts coming back, which is also why I strongly suggest not to use the autosubscription for large usergroups.

Also realize that the number of your users that will utilize this feature will not be all of them.

Alot of users HATE mailing lists. That's what this is...

However... alot of users HATE forums...

This let's you get those forum haters to participating in your forum and through the links in the bottom of the emails, gradually get them to roll around to the forum life :)

Please feel free to run this on a test server and try it out.

While the code is not yet at it's optimal form, it's definately strong enough to handle a significant load.

Dear Chris,

Thank you for PM.
I have just sent a mail just like I replied using your hack.

Sincerely,

Atsuko

Thanks!!

I'll start working on trying to process this. I found a few commands that will translate it to utf-8 so I just need to toy with it from there :)

Thanks for the response Cyricx. I don't completely understand the vbulletin mail system, so I am just trying to think of it logically. I know that when I was running 3.5 and had the auto prune usergroup modification installed, I experienced a mail backup.

It emailed users in the Unconfirmed Usergroup every 3, 5 and 7 days. Just that added load, in addition to the amount of users who subscribe to specific threads on my site, confirmation emails, comment emails, etc., I ended up having a day and half lag on sent emails.

Once I had uninstalled that modification, within a few days my mail que was back to normal.

I am looking at this as possibly doing the same thing. I may be wrong, but i wont know until I try it.

I like the concept of my staff being able to use this, so I may have to give it a go and see how it does.

Well to kinda give an idea of how the mail queue system runs from my experience dabbling in the bugger. Perhaps someone can correct me if I'm wrong.

When you post a reply with this modification, it'll run some queries to determine who all should receive an email. Then it will check to see if you have it set to run in batches. If you do have it set to run in batches, it'll add those emails to the mail queue table.

The next time a user refreshes any page on your site it'll run check how many you want to send per batch, grab those emails from the database, send them and delete those rows.

The next time it is refreshed it'll repeat this process.

I'm not one hundred percent certain on how long it waits between batches to allow another batch to be sent but I believe it's only about 5-10 seconds.

I have sent as many as 100 emails at one time with this mod prior to the vbmail days and experienced no adverse server side effects. Though I run on a VPS server. Others didn't hit problems until they tried to send 1000 emails at once! Hehe. So If you have problems with your mail backing up, try setting your batches higher, to like 10 or 20 and then monitor the system.

The reason I like the new layout is it's much more aesthetically pleasing for the forums and with the "reply box" it kinda gently encourages the users to visit the forum rather then just slink along on emails. :)

Which in the end.. is what we want... them visiting our forums. Going to the other sections of the site and heck.. in some cases... clicking on our ads :)

Hi

this would be a problem for me. I need this e-mail notification sent out without a user visiting the page to refresh it.

I have a few boards, that have not enough traffic. So maybe one user every 2 days finds the forum. Sending just the e-mail out, after 2 days of the original posting would be too late.

I know of a different forum-software (www.discusware.com), that has this e-mail reply functionality too. And they use also cronjobs for it. But I can schedule the cronjob, that because of the cronjob every 3 minutes it checks for new messages (no matter whether somebody hit the forum or not) and sends the e-mail out.

A. Could you implement this too?

B.

I need always to send out e-mails for around 500-1000 users with this for each vb-borad I have on the same server. I have several vb-boards on the same server. And each board needs to send those e-mail to this number of people for each vb-board.

Would be the mailserver/serverload because of this to be too high? I use a dedicated server.

Thanks in advance

I do have a beta script that could be potentially run from the local server on a cron job. My expertise on setting up cron jobs for a server is nil though. If you have knowledge about those I'll send you the file I have that can be run from any directory. Just PM me your email address.

As for the mailserver/serverload. I can't say for sure, depends on your server settings. I believe the individual that experienced an issue when sending 1000 emails was on a VPS, so I would venture to say that on a dedicated server you will likely have no issues.

Either way the script would not fail, only the server could. If you set up vbmail to run in batches of a few hundred though, I think you would definately be rockin strong.

UPDATED!! :)

Here's the update message I sent out -

As many have noticed I'm on a bug squashing and obliterating mission!

I had hoped to get a few more minor bugs fixed in this version but
finally found the most elusive and meanest one which was causing people to
get errors when replying to some threads.

The criteria to cause this bug was:

A user must be subscribed to the thread, the thread must be in a forum
that email integration has been enabled. That user must not be
subscribed for instant notification to the forum... only the thread.

Then you would get the error.

I've added a quick fix for the error in 2.5.4

Please upgrade.

If you notice any quirks... bugs... or issues... PLEASE let me know.
I'm devoted to making this a solid modification but I rely on your input
:)

Thank you for your support and patience!

~ Chris

Just wanted to give everyone a heads up that I hadn't dropped off the planet and I'm working on the bugs that have been reported.

Here is a list of known confirmed bugs.

These are ones that I have NOT been able to duplicate so may actual be due to issues with another modification:

• Duplicate replies when posting in a forum that EI is turned on for.
• Headers when sent through a forwarder are lost when Gmail receives them. - Not a bug! Server settings required setting the delimiter in EI options to \n
• Occasionally lists a newthread poster as "unregistered" in the email that is sent out. - FIXED in 2.5.5

These are ones that I HAVE been able to duplicate in my test board and I'm working on fixes for them now:

• Does not correctly decode iso-2022-jp emails.
• Does not carry reply title to email. - FIXED in 2.5.5
• Subscribeforums.php shows forums that are links. - FIXED in 2.5.5

Please let me know if you are also experiencing any of the unduplicated bugs so that perhaps I can compare your site to the one having the issue to determine what may be the joining factor :)

The few bugs we are down to seem to only affect one site out of 40 LOL. So the criteria for bugs is getting very obscure... but hey... like I said.. I'm committed to making the most solid major addition modification on vb.org :)

Good on you Cyricx, keep going.
vB just cannot realise how important it is.

Thanks for your hard work!

Thank you! :)

I'm going to jump straight to 2.6

Still waiting on some feedback on two obscure "bugs" so starting the coding for the other stuff that I want to get in asap.

So far coded is the user option to not receive emails for your own posts :)
Also, admins can allow users to choose if they do NOT want to be autosubscribed.

Those are now already coded and in.

Next I'm going to check out some stuff with how the delimiter is built to see if I can tighten that up. Work on the error message for the usergroup manager so that it will give you an error if you try to autosubscribe a group that is not setup to be allowed to use the feature in the forum manager and lastly... friendly involvement with the thread subscription :)

I'm a busy bee ;)

Well... I went from 2.5.4 and went straight to a 2.6 release :)

Prolly had something to do with spending 8 hours doing nothing but working on this code hehe :)

2.5.5 was created in the interim and not released as while working on some bugs I made some huge breakthroughs in what I wanted done for 2.6... which is the coexistance with thread subscriptions as well as the user options for not getting their own posts and (if the admin allows them to) users can select to not be autosubscribed.

Check my third post in this thread for details on how the coexistance with thread subscriptions work.

I'm undecided at this time if I'm going to try to work thread subscriptions to let yu reply via email or not. It could require ALOT of file editting in the functions_newpost.php file which I'm trying to avoid as many file edits as possible.

All in all, I'm excited to feel like this may be my final version for awhile with the exception of working on the last confirmed bug of it not handling japanese characters :)


Several have asked for the cron jobs not to depend upon vbulletin traffic to work.

I'm really not up for writing something that intensive... it would involve creating a queue system as well as trying to teach people how to create cron jobs on their server... yeah.. not my piece of the pie :)

Plus I'm very happy with the way that this works with vbulletin. My forums typically have 1-2 visitors on them at a time right now and it's still working great.

The point of this is NOT to replace vbulletin. I want to encourage the users to come to the forums and hopefully eventually weed them off the mailing list lifestyle and into the forum lifestyle so to speak :)

In anycase, I do have a script that could run by cron job, but it still uses vbmail which requires traffic to activate vbmail and send the emails.. so.. hey.

You could modify it to use regular php mail but I would prolly advise against it as it could crash a server if you tried to send off 1000 emails at once!

Enough babble... hope you enjoy the latest version and donations are certainly appreciated :)

Thank you!

Hello Cyricx,

I have been having a problem. Using the gmail settings you posted i get this when running from the tasks:

Warning: imap_open() [function.imap-open]: Couldn't open stream {pop.gmail.com:993/pop3/ssl}INBOX in /includes/cron/emailintegration.php on line 89

Failed to connect to the mailbox for forum - xxxxxx forum

also happens when i try selfsigned ssl pop3:

Warning: imap_open() [function.imap-open]: Couldn't open stream {pop.gmail.com:995/pop3/ssl/novalidate-cert}INBOX in /includes/cron/emailintegration.php on line 89

Failed to connect to the mailbox for forum - xxxx forum

--------------------
Now whats weird is i have POP3 on the forum settings not Imap...

Any idea what is going wrong? All the settings are right (username, pass, etc)

Make sure you have enabled pop3 in your gmail box.

Per the third post in this thread :)

• Can I use a Gmail account to pull the emails from?

You most certainly can.

Login to your mailbox at gmail, go to Settings -> Forwarding and POP/IMAP

Enable POP for the account, either from here on, or for all mail. Be sure to select archive or delete for what to do after the messages are downloaded. Otherwise the system will continue to repost the emails!!

In your forummanager use the folowing settings:

email address to reply to: username@gmail.com
hostname: pop.gmail.com
username: username@gmail.com
password: password
Type of connection: select "POP3 with SSL (Self signed certificate)"

Done!

If you don't enable your gmail box to allow people to connect to it via pop3 or imap it will give that error :)