Administrative and Maintenance Tools - Import External Images

This vBulletin 4 product imports external images in your posts from external servers or paths into a specific path, normally your forum's image folder.

Differently than the previous product Image Cache for vBulletin 3.x., it parses the posts at regular intervals for new images and replaces the external references in the background. Also, instead of using only a single folder, it creates folders for years and months, making it much easier to handle large quantities of images. The source and destination image URL's are stored into database, too, avoiding unnecessary duplicates.

It moves and optionally converts images from one place to another. It is a handy tool in order to keep your images local to your forums.

Version 1.0.4 is temporarily in BETA stage until the users have confirmed the newly requested features.
Version 1.0.5 is still in BETA stage. It adds a new bunch of additional features, one of which allows you to test your mathematical understanding ... and a few other to fine-tune CURL and socket timeouts, and resizing on the fly of too large images. If you don't know what these options do, please leave them at their default values.

[smirkley]

I am attempting to find out where it stalled out at using phpmyadmin, when I noticed something strange...

The replacement url for missing images looks like this,.. (sometimes, not all instances)

h ttp://www.mysite.com//images/misc/BrokenImage.gif
(I added a space in http to prevent this url from parsing here)

edit... figured it out, my bad, I had added a leading / to the broken image image url in acp

now just to figure out how to correctly convert it without having to edit each and every field lol

[tommythejoat]

My moderators are reporting problems with this feature. It appears that images are being replaced with the wrong, seemingly random image.

I spent some time looking at the iei_img table and the post table with the following results.

We have 798,156 posts.
23,105 of the posts contain at least one IMG bbcode.
25,859 are marked iei_parsed.

In the iei_img table we have 34,257 records.
Of these 20,420 saved images are mapping unique external images.
We have 11,015 records with our web root (forum) directory as the saved entry (no jpg)
The big problem is 911 of the save images map multiple external images with as many as 204 references down to 2 references.

Other than restoring my backups from before I installed iei, I do not know how to recover the erroneous image mappings.

I do not really understand how the batches are managed while importing images. It appears that it is not working correctly.

We are running 4.2.1 with iei 1.0.5

[smirkley]

I too have reservations on a reverse-implentation plan. Is there a method of reversing ones way out of this mod in the event it becomes undesirable in the end?

But most importantly, I wish to make this mod fully implementable as the concept is important to me.

[y2ksw]

Originally Posted by tommythejoat

My moderators are reporting problems with this feature. It appears that images are being replaced with the wrong, seemingly random image.

I spent some time looking at the iei_img table and the post table with the following results.

We have 798,156 posts.
23,105 of the posts contain at least one IMG bbcode.
25,859 are marked iei_parsed.

In the iei_img table we have 34,257 records.
Of these 20,420 saved images are mapping unique external images.
We have 11,015 records with our web root (forum) directory as the saved entry (no jpg)
The big problem is 911 of the save images map multiple external images with as many as 204 references down to 2 references.

Other than restoring my backups from before I installed iei, I do not know how to recover the erroneous image mappings.

I do not really understand how the batches are managed while importing images. It appears that it is not working correctly.

We are running 4.2.1 with iei 1.0.5

If the "missing/invalid" image replacement is disabled, the original links are maintained, when this product is unable to import an image. For this reason it is most probable that random images are still within the img tags, since those are unlikely to work, ever. For example, some sites don't allow hot linking, and thus put junk redirects on line. Then this product wants to import them and sees, it can't. If missing images are replaced, it will typically use clear.gif or something alike to replace the missing image. Else, it will leave the link intact, and it may work for a while, until this kind of images disappear from source.

There is nothing I can do about it, this is how Internet works.

You may reparse all or part of the posts setting the iei_parsed flag to 0. Some sites do allow hot linking after a while and so you may get them later. I've been using this product over many years now on large production sites and we have had a number of problems, which were all related to remote images. Some of which were WORM infected, some where hotlink protected and others just would not conform to the standards. But these are few exceptions out of thousands of imported images every month.

[y2ksw]

Originally Posted by smirkley

I too have reservations on a reverse-implentation plan. Is there a method of reversing ones way out of this mod in the event it becomes undesirable in the end?

But most importantly, I wish to make this mod fully implementable as the concept is important to me.

Eventually you could parse the old/new urls from the iei_parsed table and attempt to replace with global queries the links. On large forums this could be however a very time and resource consuming operation. And in any case, it is the only one you could use, since posts may be copied to any place and it is impossible to tell if the contained links came from your site or anywhere else.

[tommythejoat]

I think perhaps you missed my point on the multiples. I just searched my forum posts for "/2007/01/1.jpg" which should yield at most a few copies of the same image in related posts. Instead it returns 558 hits in a total of 371 posts.

I don't see any way for this to occur without an indexing error in the software. I think it is at least worth a look at the arbitrary name generator and where it stores its counter.

[y2ksw]

Originally Posted by tommythejoat

I think perhaps you missed my point on the multiples. I just searched my forum posts for "/2007/01/1.jpg" which should yield at most a few copies of the same image in related posts. Instead it returns 558 hits in a total of 371 posts.

I don't see any way for this to occur without an indexing error in the software. I think it is at least worth a look at the arbitrary name generator and where it stores its counter.

Double or more hits in a single post is possible, if for instance somebody has pasted an image N times. This happens often in my forums, where people copy their preferred smilie instead of asking me to add them to the stock items. Similarily, you'll find underlines, page separators, signatures and all kinds of graphical additions people just paste on the fly without even knowing it.

In the case of having 2 or more hits of an image in a single post, it would be interesting to see if there are actually two or more copies of that image. If it isn't, then maybe something strange is going on.

[tommythejoat]

You did notice that I said 558 copies of the same image in a total of 371 posts.

Yes it would be perfectly reasonable to see several instances of an image. It can happen when a post is quoted for example or a person might even refer to the same image in two or three different posts. When it gets to the hundreds it cannot be due to random selection by users, it can only be a bug.

If one has left the replacement image value blank as I have done, what should we expect to see? We have lots of entries in the replacement image of http://mysite/ with no file reference in the URL. I have been reading those as meaning the image was missing from the source site.

[smirkley]

I have fixed my first problem extra "/" in replacement url,... caused when I added incorrectly the leading slash in my replacement image.

Need help with my second problem.

Operation has hung and I cannot figure out how to identify the image that it is hung on.

[y2ksw]

Originally Posted by smirkley

I have fixed my first problem extra "/" in replacement url,... caused when I added incorrectly the leading slash in my replacement image.

Need help with my second problem.

Operation has hung and I cannot figure out how to identify the image that it is hung on.

It is the first post with an img tag which has not the iei_parsed flag set to 1.

SELECT * FROM post WHERE iei_parsed=0 AND pagetext LIKE '%[img%' ORDER BY postid LIMIT 1