Administrative and Maintenance Tools - Import External Images

This vBulletin 4 product imports external images in your posts from external servers or paths into a specific path, normally your forum's image folder.

Differently than the previous product Image Cache for vBulletin 3.x., it parses the posts at regular intervals for new images and replaces the external references in the background. Also, instead of using only a single folder, it creates folders for years and months, making it much easier to handle large quantities of images. The source and destination image URL's are stored into database, too, avoiding unnecessary duplicates.

It moves and optionally converts images from one place to another. It is a handy tool in order to keep your images local to your forums.

Version 1.0.4 is temporarily in BETA stage until the users have confirmed the newly requested features.
Version 1.0.5 is still in BETA stage. It adds a new bunch of additional features, one of which allows you to test your mathematical understanding ... and a few other to fine-tune CURL and socket timeouts, and resizing on the fly of too large images. If you don't know what these options do, please leave them at their default values.

[y2ksw]

Originally Posted by Vodkaholic

Yea i did manually a few times the other day still did'nt cache them tho hmm

Please turn logging of that task on, and check the logs. If an image has been imported, you can check it clicking the link in the logs ... those images should be then stored locally.

After checking, disable the logging because it may flood your logs with a tremedous number of entries, depending on the number of pasted images ...

And of course, tasks are not running until forums are on.

[BirdOPrey5]

Can you explain more about the "cache" option in the new Beta?

[y2ksw]

Originally Posted by BirdOPrey5

Can you explain more about the "cache" option in the new Beta?

Not much more than it says in the options. If the cache option is on (default), all image references are loaded into memory by a single query (and N fetches), if it is off, every image is individually searched in the database. Thus, if you have a large forum and a huge amount of images, PHP may run out of memory and the mod stops apparently to work ("could not allocate N bytes" error). In this case you have 2 options: empty the iei_img table, or switch caching off.

[BirdOPrey5]

Any estimate on what you'd consider a "huge amount" of images? 100,000? 1,000.000? 10,000,000? More than that?

[y2ksw]

Originally Posted by BirdOPrey5

Any estimate on what you'd consider a "huge amount" of images? 100,000? 1,000.000? 10,000,000? More than that?

That depends on the memory you are using for each PHP process and the length of the URL's. When the plugin stops working from one day to the other, you probably have reached this limit.

On one site we have 120.000+ images and it continues to work, thus maybe that's not huge. But we also give unlimited memory to the PHP processes and the server can hold up to 16 GB. Maybe a million is huge, in this context

[BirdOPrey5]

Thanks.

[BadgerDog]

Went live today with 1.04 .... :up:

Crossed my fingers and opened up most of our forums to start importing 10 posts at a time from the oldest to newest ....

First few runs seemed quicker with this ordering (old to new) than the testing we did with (new to old)... also turned OFF logging after first few runs ...

Let's see how long it takes to get through our site to the latest ...

y2ksw and BirdOPrey5 ... thanks for all the help from both of you ...

Regards,
Doug

Update: Ok, I started the live execution yesterday at about 4:00 pm (same time as this original post) and checked the first few logs. Everything seemed to be working according to the logs and the earliest threads seemed to be processed. Out of curiosity to see where it was at this morning, I turned LOGGING back ON and waited until cycle for cron had past. I checked logs and there wasn't anything there at all. So, I manually ran the process and watched it (see pic attached). It's been sitting like that for almost 15 minutes, just supposedly processing 10 posts, so I suspect it's hung and I have no idea how long it hasn't been running since I started it at 4:00 pm yesterday. I also don't know if there's been any database damage or not due to the hanging, but the vBulletin error logs don't show any SQL errors and the board seems to be running fine.

I assume I should quite the processing on the screen attached and do what next?

Update to update: In checking actually file storage on the server under /imported, it appears that the last pics imported happened at about 5:05 pm, so it appears that the import mod ran just over an hour after I had originally started it before it stopped working .... and hasn't worked since.

Another update: I have tried turning OFF the cache and that didn't make any difference.

Another update: For info purposes, we are running on a dedicated private host and Safe Mode is OFF, so that shouldn't be a time out problem. y2ksw suspects the job is timing out, so I've asked him for some debug code to be built into a test version of the cron job for manual run testing, so we can see where this is failing and perhaps why.

[PhilG]

I have just tested this and it's got a serious bug.

I have set "Forum URL replacement" to http://static.domain.com and alot of images get replaced with just that domain. For example there was an image "http://www.angelfire.com/oz/gbeze/images/circuit.gif" and it got replaced with "http://static.domain.com/

Apart from that great mod.

[y2ksw]

Originally Posted by PhilG

I have just tested this and it's got a serious bug.

I have set "Forum URL replacement" to http://static.domain.com and alot of images get replaced with just that domain. For example there was an image "http://www.angelfire.com/oz/gbeze/images/circuit.gif" and it got replaced with "http://static.domain.com/

Apart from that great mod.

Is your forum path starting with http://static.domain.com ? In other words, is your forum contained in the forum replacement or vice versa?

Could you please attach a setting with which I may reproduce this behaviour?

Thanks

[sarangan]

It would be better if there was an option to exclude forums instead of include forums. :-) Looking forward to have that option also in the feature version. A super and nice plugin. I love it.. Thanks a lot friend.. :-)