Administrative and Maintenance Tools - Import External Images

This vBulletin 4 product imports external images in your posts from external servers or paths into a specific path, normally your forum's image folder.

Differently than the previous product Image Cache for vBulletin 3.x., it parses the posts at regular intervals for new images and replaces the external references in the background. Also, instead of using only a single folder, it creates folders for years and months, making it much easier to handle large quantities of images. The source and destination image URL's are stored into database, too, avoiding unnecessary duplicates.

It moves and optionally converts images from one place to another. It is a handy tool in order to keep your images local to your forums.

Version 1.0.4 is temporarily in BETA stage until the users have confirmed the newly requested features.
Version 1.0.5 is still in BETA stage. It adds a new bunch of additional features, one of which allows you to test your mathematical understanding ... and a few other to fine-tune CURL and socket timeouts, and resizing on the fly of too large images. If you don't know what these options do, please leave them at their default values.

[PhilG]

Originally Posted by y2ksw

Is your forum path starting with http://static.domain.com ? In other words, is your forum contained in the forum replacement or vice versa?

Could you please attach a setting with which I may reproduce this behaviour?

Thanks

I have attached the settings.

I have the main forum setup on www.domain.com and images get served from static.domain.com.

Edit: It seems to only happen to images that are broken/missing.

Regards,
Phil

[BadgerDog]

Update:

With thanks to y2ksw for building in some debug code to a special beta version to troubleshoot problems with this mod, we have been able to isolate the issues that are causing hangs (cron lock-ups), which we and others may be experiencing. He's pretty busy at work, so suggested that we post our findings here so that possibly programmers such as BirdOPrey5 or others, might be able to help us figure out how to fix the lock up issue.

It appears that the mod locks up when it encounters pics referred to with the IMG tag from other sites such as photobucket etc., that are 1024x768 pixels or larger. I original thought that it also might have something to do with byte size as well, since we do have some inconsiderate posters who IMG link to external pics that are 1 Mb and even up to 3 Mb in size.

However, after checking over over 100 pics that the mod failed and hung-up on, there are a lot that are reasonably sized (100k to 300k), but are over 1024x768 in pixel size, so my assumption is the hangs are caused by the handling of pixel sizes as opposed to file sizes. The 1024x768 number is a guess, purely based upon the fact that the mod seems to process pics with that pixel width and height, but we haven't been able to find ANY pic over that sizing that hasn't locked up the mod.

We've been able to manually circumvent the problem by running the cron job manually a forum at a time. Once it hangs, we examine the Cron Error Log and look at the code. It looks like this at the point it's locked up and stopped processing:

=======================================
vB4 Import External Images
2011-02-16 08:00:02 345 Get forum ID's
2011-02-16 08:00:02 349 Get image tags
2011-02-16 08:00:02 358 Adjust options
2011-02-16 08:00:02 379 Cache registered image URL's
2011-02-16 08:00:02 177 SELECT oldurl, newurl
FROM iei_img
2011-02-16 08:00:02 386 For each record with images ...
2011-02-16 08:00:02 389 Include an/or exclude specific forum sections
2011-02-16 08:00:02 409 SELECT postid, p.dateline, pagetext
FROM post p
INNER JOIN thread t ON p.threadid = t.threadid
WHERE iei_parsed=0
AND forumid IN (72)

ORDER BY postid
LIMIT 0, 10
2011-02-16 08:00:02 413 Post: 61775
2011-02-16 08:00:02 417 Find all embedded images
2011-02-16 08:00:02 420 Searching for tag: img
2011-02-16 08:00:02 430 Search for images to be ignored
2011-02-16 08:00:02 445 If we haven't got the image yet, download
2011-02-16 08:00:02 453 Pseudo-loop
2011-02-16 08:00:02 456 Get file from remote site
2011-02-16 08:00:02 285 Use CURL for: http://www.thefirearmblog.com/blog/w...compressed.jpg
2011-02-16 08:00:03 324 Content size: 788141
2011-02-16 08:00:03 464 Get file extension
2011-02-16 08:00:03 467 Make path
2011-02-16 08:00:03 471 Get file name and/or index
2011-02-16 08:00:03 139 Get only the file name (no directory info)
2011-02-16 08:00:03 142 Strip out non alpha-numeric characters
2011-02-16 08:00:03 145 Set max file name length to 40 characters
2011-02-16 08:00:03 148 Increment number if file exists
2011-02-16 08:00:03 481 See if we have valid contents
===========================================

We check the post ID, call it up on the forums and invariably, it's an IMG reference to an off-site pic that's bigger than 1024x768 pixels.

We then SAVE the pic to our local drive. Delete the reference in the post to it. Use the vBulletin advanced ATTACH javascript to upload it to the post as an attachment and SAVE the edited post.

We re-start the cron job and it now moves past this post and proceeds normally until it runs into another post with an IMG referenced pic larger than 1024x768.

Using this workaround, we've been able to work through some of our forums, but it takes 6-8 hours of manual work to do so.

So, the question is, can someone help us by looking at the code and figuring out where *and why) it's locking up on these 1024x768 pics? Perhaps it needs a pixel re-sizer function built in front of it, BEFORE it begins to import the pic?

Second problem ...

We discovered an anomaly as to file name conventions (see attached pic). It appears that in some cases the mod imports and stores the pics, but for some reason it attaches ?t=xxxxxx or ?a=xxxxx after the .jpg extension (see attached pic). In the actual post, the filename is the same way, but of course the pic doesn't show.

If I manually rename the file in both the offending post and the associated /includes directory, it fixes the problem and the pics appear in the post. Unfortunately, so far, I can't find a simple way to locate all the filenames on the server that are messed up and also find them in the actual forums so I can start manually fixing posts.

This one y2ksw says is fixable by turning ON the Always Convert Images option. Y2ksw explained why this would stop the problem from occurring in future, but not being as technical as he is, I didn't quite understand it. We're still trying to figure how to locate all the pics that have been processed already and perhaps some kind of automated handling to adjust the file names on both the server and the associated posts, so they actually appear.

Anyway, this is an excellent mod, particularly for a research site such as ours, to enable us to preserve old posts and pics for posterity. With just a bit more de-bugging and sorting out this critical hanging problem, it would be perfect....

Hope this helps other users here who may be considering using this mod for their sites, particularly larger ones with lots of pics.

Regards,
Doug

[BirdOPrey5]

I can confirm larger images aren't importing right, such as this one:
http://www.qapla.com/ju/DSC00261.JPG

Might be a sever limitation, not sure... will see if I can figure anything out.

Can you give me a link to an image that is giving you the second problem?

[dacho]

I have two questions please:
1. Is it possible to import the images to sub-domains or other domain?
2. If the files are not saved in my database, can also add a watermark option on imported images?

[BadgerDog]

Originally Posted by BirdOPrey5

I can confirm larger images aren't importing right, such as this one:
http://www.qapla.com/ju/DSC00261.JPG

Might be a sever limitation, not sure... will see if I can figure anything out.

Can you give me a link to an image that is giving you the second problem?

Stand by ...

I think you may be onto something ...

y2ksw sent me an email and suggested I try to increase my PHP memory variable from the current 16 MB, which he thought was maybe is a little low. He recommended to give it at least 8 MB more (on linux, this setting (memory_limit) should be at /etc/php5/php.ini). I've asked our tech support to increase it from 16MB to 32MB just in case.

Once they do that, I'll start re-testing the big pics....

Thanks for your testing to validate the issue ... :up:

Regards,
Doug

[BadgerDog]

Update:

Changing PHP memory_limit variable from 16 MB to 32 MB worked great ... :up:

Fixed 90% of the hangs, however still chokes and locks up mod on any pic thats 2000x1600 referenced through IMG tag. I wish there was some automated way of recognizing members are IMG linking to pics over a certain size as it's happening, stop the process and send them an error message asking them to resize their pics.

I can't stop members from linking to those as many of our members aren't really that computer literate (most 60+ years old) so I'm going to ask our tech support to increase the PHP memory_limit variable once again, this time from 32 MB to 48 MB. I suspect that will work for the latter large pixel sizes.

Will advise when I've done that and re-tested ...

Thanks ..

Regards,
Doug

[BadgerDog]

Final Update:

Problem solved ... mod working fine now with one limitation

I further increased the PHP memory again, this time from 32 MB to 64 MB....

I then tried to once again process a pic that was 2000x1600 pixels and it still hangs, so it seems that if the size is as large as this, even 64 MB of PHP memory is not enough to handle it.

I've decided to not bother trying to automate using this mod to handle any 2000x1600 pixel sized pics, opting instead to try to discourage members from linking with IMG to them, or if they do link to them, manually circumventing those posts with this mod by manually resizing the member's pics.

Is there some way of checking in advance (even a separate mod) to stop people from linking via IMG's to pics larger than a certain pixel size?

Thanks again to y2ksw and BirdOPrey5 for all the help and guidance in getting this mod going for our site. :up:

Regards,
Doug

[BirdOPrey5]

In my admittedly quick search I couldn't find any way of finding the image dimensions without downloading the full image- and if that is what is causing the problem checking what we get would not be a solution- if it's too big it will bug out anyway.

[BadgerDog]

Originally Posted by BirdOPrey5

In my admittedly quick search I couldn't find any way of finding the image dimensions without downloading the full image- and if that is what is causing the problem checking what we get would not be a solution- if it's too big it will bug out anyway.

I agree... a chicken and egg syndrome...

I wonder if there's any javascript or other already written query type code that can be run against a graphic file, in order to test for embedded pic data or something else, without having to download the whole file...

Regards,
Doug

[d19rnyxx]

I needed to know if this mod has any issues with vbOptimise running memcache
Thanks