I have attached the settings.

I have the main forum setup on www.domain.com and images get served from static.domain.com.

Edit: It seems to only happen to images that are broken/missing.

Regards,
Phil

Update:

With thanks to y2ksw for building in some debug code to a special beta version to troubleshoot problems with this mod, we have been able to isolate the issues that are causing hangs (cron lock-ups), which we and others may be experiencing. He's pretty busy at work, so suggested that we post our findings here so that possibly programmers such as BirdOPrey5 or others, might be able to help us figure out how to fix the lock up issue.

It appears that the mod locks up when it encounters pics referred to with the IMG tag from other sites such as photobucket etc., that are 1024x768 pixels or larger. I original thought that it also might have something to do with byte size as well, since we do have some inconsiderate posters who IMG link to external pics that are 1 Mb and even up to 3 Mb in size.

However, after checking over over 100 pics that the mod failed and hung-up on, there are a lot that are reasonably sized (100k to 300k), but are over 1024x768 in pixel size, so my assumption is the hangs are caused by the handling of pixel sizes as opposed to file sizes. The 1024x768 number is a guess, purely based upon the fact that the mod seems to process pics with that pixel width and height, but we haven't been able to find ANY pic over that sizing that hasn't locked up the mod.

We've been able to manually circumvent the problem by running the cron job manually a forum at a time. Once it hangs, we examine the Cron Error Log and look at the code. It looks like this at the point it's locked up and stopped processing:

We check the post ID, call it up on the forums and invariably, it's an IMG reference to an off-site pic that's bigger than 1024x768 pixels.

We then SAVE the pic to our local drive. Delete the reference in the post to it. Use the vBulletin advanced ATTACH javascript to upload it to the post as an attachment and SAVE the edited post.

We re-start the cron job and it now moves past this post and proceeds normally until it runs into another post with an IMG referenced pic larger than 1024x768.

Using this workaround, we've been able to work through some of our forums, but it takes 6-8 hours of manual work to do so.

So, the question is, can someone help us by looking at the code and figuring out where *and why) it's locking up on these 1024x768 pics? Perhaps it needs a pixel re-sizer function built in front of it, BEFORE it begins to import the pic?

Second problem ...

We discovered an anomaly as to file name conventions (see attached pic). It appears that in some cases the mod imports and stores the pics, but for some reason it attaches ?t=xxxxxx or ?a=xxxxx after the .jpg extension (see attached pic). In the actual post, the filename is the same way, but of course the pic doesn't show.

If I manually rename the file in both the offending post and the associated /includes directory, it fixes the problem and the pics appear in the post. Unfortunately, so far, I can't find a simple way to locate all the filenames on the server that are messed up and also find them in the actual forums so I can start manually fixing posts.

This one y2ksw says is fixable by turning ON the Always Convert Images option. Y2ksw explained why this would stop the problem from occurring in future, but not being as technical as he is, I didn't quite understand it. We're still trying to figure how to locate all the pics that have been processed already and perhaps some kind of automated handling to adjust the file names on both the server and the associated posts, so they actually appear.

Anyway, this is an excellent mod, particularly for a research site such as ours, to enable us to preserve old posts and pics for posterity. With just a bit more de-bugging and sorting out this critical hanging problem, it would be perfect....

Hope this helps other users here who may be considering using this mod for their sites, particularly larger ones with lots of pics.

Regards,
Doug

I can confirm larger images aren't importing right, such as this one:
http://www.qapla.com/ju/DSC00261.JPG

Might be a sever limitation, not sure... will see if I can figure anything out.

Can you give me a link to an image that is giving you the second problem?

I have two questions please:
1. Is it possible to import the images to sub-domains or other domain?
2. If the files are not saved in my database, can also add a watermark option on imported images?

Stand by ... :)

I think you may be onto something ... ;)

y2ksw sent me an email and suggested I try to increase my PHP memory variable from the current 16 MB, which he thought was maybe is a little low. He recommended to give it at least 8 MB more (on linux, this setting (memory_limit) should be at /etc/php5/php.ini). I've asked our tech support to increase it from 16MB to 32MB just in case.

Once they do that, I'll start re-testing the big pics....

Thanks for your testing to validate the issue ... :up:

Regards,
Doug

Update:

Changing PHP memory_limit variable from 16 MB to 32 MB worked great ... :up:

Fixed 90% of the hangs, however still chokes and locks up mod on any pic thats 2000x1600 referenced through IMG tag. I wish there was some automated way of recognizing members are IMG linking to pics over a certain size as it's happening, stop the process and send them an error message asking them to resize their pics.

I can't stop members from linking to those as many of our members aren't really that computer literate (most 60+ years old) so I'm going to ask our tech support to increase the PHP memory_limit variable once again, this time from 32 MB to 48 MB. I suspect that will work for the latter large pixel sizes.

Will advise when I've done that and re-tested ...

Thanks .. :)

Regards,
Doug

Final Update:

Problem solved ... mod working fine now with one limitation

I further increased the PHP memory again, this time from 32 MB to 64 MB....

I then tried to once again process a pic that was 2000x1600 pixels and it still hangs, so it seems that if the size is as large as this, even 64 MB of PHP memory is not enough to handle it.

I've decided to not bother trying to automate using this mod to handle any 2000x1600 pixel sized pics, opting instead to try to discourage members from linking with IMG to them, or if they do link to them, manually circumventing those posts with this mod by manually resizing the member's pics.

Is there some way of checking in advance (even a separate mod) to stop people from linking via IMG's to pics larger than a certain pixel size?

Thanks again to y2ksw and BirdOPrey5 for all the help and guidance in getting this mod going for our site. :up:

Regards,
Doug

In my admittedly quick search I couldn't find any way of finding the image dimensions without downloading the full image- and if that is what is causing the problem checking what we get would not be a solution- if it's too big it will bug out anyway.

I agree... a chicken and egg syndrome... :D

I wonder if there's any javascript or other already written query type code that can be run against a graphic file, in order to test for embedded pic data or something else, without having to download the whole file...

Regards,
Doug

I needed to know if this mod has any issues with vbOptimise running memcache
Thanks

None I can think of.

Greta mod and I've been looking for something like this for a while.

I don't want to grab every image that is posted to my forum, just the ones worthy of keeping. What I want to do is

1.) Create some archive forums for storing threads that are worthy of retention

2.) My archives are structured under one Category called Archives. Under that there are 7 or 8 other Sub-categories for historical eras. Under each sub-category there are 3 sub-forums.

3.) only Admins will be allowed to post in the Archives

4.) When Admin move a thread from normal posting forums to the 'Archive' forums all external images are then 'grabbed' and moved to localhost store (either in database or file system). This would mean the mod would have to treat a moved thread as a new post.

In my opinion this would make the perfect archiving solution and would mean I wasn't saving every piece of crud my users posted to the forums, just the stuff that needs saving. Yes, there's a manual element to it but then archiving is very much a manual process. So the ability to exclude all forums and set specific forums would be essential in this process.

Rgds

Pete

Yeah, I agree :)

The philosophy of this plugin is however less archiving than saving images from disappearing over time. However, you may change the parsed sections and subfolders from time to time and achieve then a kind of archiving feature as you describe.

Ah....but maybe I've given you something to think about? :)

At any rate I shall be installing this on my test rig this weekend and having a play, thanks for making it available Giovanni!

HI,

is it possible to import the images to other domain?

perfect mod,is there way to add Watermark Image at specific position

Short answer: no :)

You would like to store the images to a media server which would need a lot of more operations. However, if you have Linux, you can use the fsftp file system in order to move imported files into another server, or a symbolic link to move it on the same server to another domain. Together with the Forum URL Replacement option, you may achieve then the desired effect.

Watermarking is beyond the idea of this product.

I feel it's also a bit unethical to sign images as if they were yours, when they have been taken from an external site, instead. With adding watermarking, I would need to add special policies to my software in order to ensure that copyright infringement wouldn't harm me, in case you get sued. Depending on the laws in the country of the persuer, the software producer may have responsibilities regarding copyright infringements.

first of all thank you for your good work

secondly I have a giant forum (over 1 million posts) and I love to install your hack, I am thinking of a small script file that will do the operation of moving uploaded files to multiple domains to spread the files and reduce the multi million files to be congested in single hosting account.

I'll let you know of what result I am getting into :)

thanks again

I have this error for some images:

Database error in vBulletin 4.1.2:

Invalid SQL:
INSERT IGNORE INTO iei_img (
oldurl,
newurl
) VALUES (
'http://.....com/.21847008.lO70/christmasposter5.jpg',
'http://....net/images/imported/2010/12/6.jpg'
);

MySQL Error : MySQL server has gone away
Error Number : 2006
Request Date : Saturday, March 19th 2011 @ 07:08:56 PM
Error Date : Saturday, March 19th 2011 @ 07:10:43 PM
Script : http://..../cronadmin.php?do=runcron&cronid=90
Referrer : http://..../cronadmin.php?do=runcron&cronid=90

I also vote for CMS post. Thank you for the nice mod. Now find the way to solve database error.

Thanks for this great AddOn, and i also vote for including the CMS

not all watermark will be illegal , on my side i will use watermark with :
hosted by mysite.com
that is it
so it depend on how is the jpg wich used in watermark not the software

Thank you for update

but i have Database error

any help plzzz

Sorry, but this error does not come from my plugin. I don't use vbulletin attachments, but a completely different table (iei_img) for filename storage.

I do understand your concerns, but legal watermarking or not, I would have to protect myself and I don't like it ;)

this erorr shows after i upgrade

mybe not from your plugin but mybe yes i dont know how that hapend or mybe an conflicted with other plugin
anyways thank you bro

very very good idea. thank you for sharing this mod!

Did you decide if you would make this work for the CMS as well? Thanks!

kether

i imported my vbulletin from a old Snitz forum which had a different tag for image.
Would this product scan also those images?
All images link can be scanned?
And what about the absolute path images files such as ../public/data/user/image.jpg ?
Will this be scanned and substituted by a valid link? (they are no more valid link because the path is changed or vb doesnt recognize them)

not yet :)

As far as I recall, converted Snitz forums require HTML enabled and thus have all (external) images directly embedded. If you have different BB Code tags for images, you may try to add them to the options in order to import also those. However, it is a feature you must try first in a test bed.
Absolute paths are not recognized and you should convert those paths into valid URL's first, anyway.

:) i feel like we talked previously in another language..

Thanks for your reply.
My snitz tag works in VB, but the problems are where images are embedded with an absolute path

This could be great. But how to do that? A "search and replace" somewhere? Or a product?

Probably you will need to use a query, which replaces all absolute references accordingly to your new board settings:

and copy or move also your images to the new relative path you define here. Please make sure to backup your post table in case your query does not work as expected.

thanks a lot.
I am becoming friendly with products and plugin but I still pay attention to DB modification :)
I presume i have to back up the db table (by the way, i added the auto bkp thanks to a mod here) , do the query on the table and put it back in the db

first of all great work and appreciated ...
Second,
I downloaded the Mod and i created images/uploads folder with permission or 777 and i uploaded the PHP file to the includes/cron also i copied the file to includes direct ... i uploaded the the plugin and enabled it i found new folder created in the uploads --> 2011 --> 4 but nothing inside it ... till the moment nothing happened and no single image imported?!!!!!
is the issue in defining the tag which in the options of this mod which is by default img do i need to change it or what is the issue exactly!!!!!
my server:
===
Operating System:
CentOS 5
RAM:
2 GB
Total Disk Space:
30 GB
Bandwidth Quota:
1000 GB
===
or do i need to create any table in the DB ... or some permissions from the server is required for this hack?!!

You may need to enable fopen() to access external URL's or install CURL and allow that it accesses external content. Since it is your server importing the images, it needs also access to other sites. This may be a firewall rule and/or a permission in your php.ini file (search for fopen or URL).

thanks for your response the property allow_url_fopen is on ... what will be other issues which prevents this hack from working ...
how i can catch the error exactly since every thing seems ok but still it is not working ...
is it becuase i used the latest version of the hack 1.0.5 ...

any support please ...

What's about the firewall?

If you create a test.php with:
does it show you something? (You should see the google page at least to some extent)

I created this php file and I saw google page ...
so is it a matter of the firewall ... i checked my server, and curl is already installed ....
note some of the posts was having not exists images now it has been replaced by the error image which i defined in the hack options ...
is it a matter of time outs in the hack settings? ...
or settings of the firewall?

If you ?have seen the google page, you firewall is open. So maybe you have images which cannot be imported for some reason. Do you have this product running for a long time yet? If you have for example 1 image each 1000's post, it may take quite some time before something useful happens. Are there any entries in the iei_img table?