Administrative and Maintenance Tools - Import External Images

This vBulletin 4 product imports external images in your posts from external servers or paths into a specific path, normally your forum's image folder.

Differently than the previous product Image Cache for vBulletin 3.x., it parses the posts at regular intervals for new images and replaces the external references in the background. Also, instead of using only a single folder, it creates folders for years and months, making it much easier to handle large quantities of images. The source and destination image URL's are stored into database, too, avoiding unnecessary duplicates.

It moves and optionally converts images from one place to another. It is a handy tool in order to keep your images local to your forums.

Version 1.0.4 is temporarily in BETA stage until the users have confirmed the newly requested features.
Version 1.0.5 is still in BETA stage. It adds a new bunch of additional features, one of which allows you to test your mathematical understanding ... and a few other to fine-tune CURL and socket timeouts, and resizing on the fly of too large images. If you don't know what these options do, please leave them at their default values.

[mescalin]

I there a planned update for a cronjob for deleting unused files?

[y2ksw]

Originally Posted by oldlock

Does the resize function in the mod force the creation of thumbnails in the normal vb manner ? If not is there anyway to create thumbnails and place them in the posts as the importation takes place ?

No. This plugin imports only images and stores them on the file system instead of hot-linking.

Originally Posted by mescalin

I there a planned update for a cronjob for deleting unused files?

Yes. But I have no idea when.

[y2ksw]

*** FOR LINUX EXPERTS ONLY ***

WARNING: If you don't trust me (you shouldn't) or yourself, please try this first in a virtual box or fail-proof test environment. It is entirely up to you to verify this working properly. If you are not a Linux expert, you probably are better off with asking some help!

Many users of this plugin have asked, how to move their images from one place to another, preferably by using a FTP connection to their media server. This guide will show you how to install a local mount point to a remote server via FTP. I have tried this on two Linux Ubuntu 10.04 LTS production boxes, one holding the mount and the other keeping the images and, mainly, large videos.

All you need is a media server (may be a Windows server, as well) at some place with a FTP connection, and a Linux web server. On Linux Ubuntu you would issue a:

# aptitude install curlftpfs

but on other systems you may use apt-get, yum or rpm. All what matters is installing curlftpfs.

Once installed, you would create typically a new directory, but for moving files from one place to another, you would use just "the one" folder you already have, for example:

/var/www/domain.tld/htdocs/images/imported

The imported folder would typically hold hundreds or thousands of large files, which you would like to move elsewhere. First, you would now move or copy all files to the new server destination, and then create the FTP mount point as follows:

# curlftpfs -o allow_other,nonempty ftp_user:ftp_password@your.media.server/anywhere/you/want/ /var/www/domain.tld/htdocs/images/imported

If you have an empty folder now, you would not need to specify nonempty, thus issue:

# curlftpfs -o allow_other ftp_user:ftp_password@your.media.server/anywhere/you/want/ /var/www/domain.tld/htdocs/images/imported

ftp_user is the username and ftp_password the password to connect to your media server. Since passwords are clear, make sure to use passwords different from your root account "here" and/or "there". your.media.server would be your host name for all the files and folders, and an optional /anywhere/you/want/ would indicate a subfolder on the "other" machine. /var/www/domain.tld/htdocs/images/imported indicates the mount point. It makes no difference with or without a leading slash of your folders.

If everything went right, you are now able to play with your FTP connection, which essentially behaves as a normal folder:

# cd /var/www/domain.tld/htdocs/images/imported
# ls -lA
# mkdir abc
# touch abc.txt
# ls -lA
# df -h
# umount /var/www/domain.tld/htdocs/images/imported
# mount /var/www/domain.tld/htdocs/images/imported

and so on. Somebody now starts to dream ... what a great connection

Now, we have to add this feature to the fstab in order to have this feature always available. If the remote server isn't available at (re)boot time of this server, obviously the mount will not work. We open /etc/fstab and add this line at the end:

curlftpfs#ftp://ftp_user:ftp_password@your.med...here/you/want/ /var/www/domain.tld/htdocs/images/imported fuse defaults 0 0

Now, you would typically add a redirect to your .htaccess file from your normal image folder to your media server. Of course, it must have a web server, too, eventually at the specified location:

RedirectMatch ^/images/imported/(.*) http://your.media.server/anywhere/you/want/$1

At your media server you may monitor the new function by watching your log, for example:

# tail -f /var/www/your.media.server/weblogs/access.log

curlftpfs offers a whole lot of additional features, including SSH, proxy and write protection, which may e useful in come contexts. Also, please make sure to exclude the mount point from your round-robin rsync or your backup, since rsync and backup may pull in a lot more traffic than expected.

[mescalin]

My users are adding some links like 11234.jpg?uuygyug

This question mark after the variable prevents mod from importing. it seems as imported but no such file found. So images goes missing. Any idea about fixing this?

Edit: ? in filename becomes .jpg%3 but in the post link to image is still like .jpg?

Thank you..

[y2ksw]

Originally Posted by mescalin

My users are adding some links like 11234.jpg?uuygyug

This question mark after the variable prevents mod from importing. it seems as imported but no such file found. So images goes missing. Any idea about fixing this?


Edit: ? in filename becomes .jpg%3 but in the post link to image is still like .jpg?

Thank you..

Enable the "Always Convert Images" option. This way, all images are converted to jpg files and accordingly renamed.

How filenames are translated depend on the operating system. On Linux for example, the ? symbol remains, but still could not be found, because ? is a HTML query on most systems.

[mescalin]

i thought about renaming and converting options but if i convert all images animated gifs will not work i guess?

if i enable rename files option only, can it fix that problem?

[y2ksw]

Originally Posted by mescalin

i thought about renaming and converting options but if i convert all images animated gifs will not work i guess?

if i enable rename files option only, can it fix that problem?

This will do (= Use Original File Names: No)

[mescalin]

its me again..

in my forums some pics are replaced with wrong images that are allready in imported folder. Product does not import these images but replaces them with nonrelated existing pics.

i tried several different settings but couldnt prevent that behaviour. Any idea what can i do?

Ps its a rare behaviour. I have about 1,5m posts but it doesnt happen very often..

[rgreene1971]

Would it be possible to use this to import images from an old smf 2.0.2 forum? The old forum is still on my server at location www.mysite.com/forums with my new running vb4 forum at www.mysite.com/vbforum

This would be a great way to slowly import all my images from the old site

[y2ksw]

Originally Posted by mescalin

its me again..

in my forums some pics are replaced with wrong images that are allready in imported folder. Product does not import these images but replaces them with nonrelated existing pics.

i tried several different settings but couldnt prevent that behaviour. Any idea what can i do?

Ps its a rare behaviour. I have about 1,5m posts but it doesnt happen very often..

I believe this may be a DNS related problem.