Administrative and Maintenance Tools - Import External Images

This vBulletin 4 product imports external images in your posts from external servers or paths into a specific path, normally your forum's image folder.

Differently than the previous product Image Cache for vBulletin 3.x., it parses the posts at regular intervals for new images and replaces the external references in the background. Also, instead of using only a single folder, it creates folders for years and months, making it much easier to handle large quantities of images. The source and destination image URL's are stored into database, too, avoiding unnecessary duplicates.

It moves and optionally converts images from one place to another. It is a handy tool in order to keep your images local to your forums.

Version 1.0.4 is temporarily in BETA stage until the users have confirmed the newly requested features.
Version 1.0.5 is still in BETA stage. It adds a new bunch of additional features, one of which allows you to test your mathematical understanding ... and a few other to fine-tune CURL and socket timeouts, and resizing on the fly of too large images. If you don't know what these options do, please leave them at their default values.

[gerryvz]

In addition, here is the error I received a few times relating to the database timing out. What changes can I make to the settings to avoid this?

Database error in vBulletin 4.2.4:

Invalid SQL:
INSERT IGNORE INTO vb_iei_img (
oldurl,
newurl
) VALUES (
'http://217.118.128.96/charlesironside/FMPro?-db=ecar2dta.fp5&key=41457&-img',
'https://www.500eboard.com/forums/clear.gif'
);

MySQL Error : MySQL server has gone away
Error Number : 2006
Request Date : Friday, November 17th 2017 @ 11:43:51 AM
Error Date : Friday, November 17th 2017 @ 11:44:52 AM
Script : http://www.500eboard.com/forums/admi...ron&cronid=171
Referrer : https://www.500eboard.com/forums/adm....php?do=modify
IP Address : 100.16.101.179
Username : gerryvz
Classname : vB_Database_MySQLi
MySQL Version :

[y2ksw]

Originally Posted by gerryvz

Hi Giovanni,

Great mod, and thank you so much for creating it. I am running 4.2.4 and installed the mod yesterday. I have around 7,300 users and 150,000 total posts, with a relatively small number of overall posts that would require images to be downloaded (I would say the majority of images on my forum are directly attached and residing already in the file system).

Server - Linux + Apache
vbulletin - 4.2.4
php - 5.6.31
php max post size - 64MB
php max upload size - 64MB
php memory limit - 256MB
mySQL - 5.5.7
mySQL packet size - 10MB

The mod started downloading and it got through about 4 of the 9 years of posts on my forum. Now it has frozen since last night. I turned on the logging this morning and only came across this single entry in the log since I turned it on, which is attached. The offending post has some images attached to it, using the external host home.btconnect.com

The images seem to be small in size and pixels, and I downloaded them manually and attached them to the post to see if I could have the mod continue downloading other posts. So far, I haven't been able to get it started again.

Attached is the error, as well as my settings screens.

Also, I have now received two times the mySQL error as attached, which I read appears to be the database server "timing out."

The mod is definitely enabled and running, download folder and sub-folders are 777 permissions, etc.

Does it look to you that there is an image that is too large, or do you have other ideas as to what is causing the mod to stop indexing and downloading?

EDIT: A new post that was made by one of my users today was indexed and the external images were replaced, so I know the code is working. However, it seems stalled and won't continue indexing beyond a certain point in the past. It must be hanging up on something that is keeping it from continuing past indexing. Please advise

SECOND EDIT: I emptied the table in the database (it has a few more than 3,000 records in it) and the code executed again, but then stopped, and it had seven new records in the database. I emptied the table again and it sent through one more cycle of downloading a few more images, and putting seven new records in the database again.

So, something related to the database and/or php memory seems to be preventing the code from executing correctly.

Any thoughts?

Thanks
Gerry






Please try to disable the URL Cache. Then, all research moves from memory to database, a bit slower but less memory hungry.

[y2ksw]

Originally Posted by gerryvz

In addition, here is the error I received a few times relating to the database timing out. What changes can I make to the settings to avoid this?

Database error in vBulletin 4.2.4:

Invalid SQL:
INSERT IGNORE INTO vb_iei_img (
oldurl,
newurl
) VALUES (
'http://217.118.128.96/charlesironside/FMPro?-db=ecar2dta.fp5&key=41457&-img',
'https://www.500eboard.com/forums/clear.gif'
);

MySQL Error : MySQL server has gone away
Error Number : 2006
Request Date : Friday, November 17th 2017 @ 11:43:51 AM
Error Date : Friday, November 17th 2017 @ 11:44:52 AM
Script : http://www.500eboard.com/forums/admi...ron&cronid=171
Referrer : https://www.500eboard.com/forums/adm....php?do=modify
IP Address : 100.16.101.179
Username : gerryvz
Classname : vB_Database_MySQLi
MySQL Version :

The database server connection times out before the page or script finishes. This should never occur, but quite some hosting services "enforce" this policy in order to keep up with thousands of sites in a box. In a normal server, the connections time out once a day or so.

If this is your box, please make sure that the mysql server never drops connections, but also disallow permanent connections. The speed loss is won back by making less connections due to timed out pages and user refreshes.

[gerryvz]

Hi Giovanni,

Thank you for your responses. Rarely is a developer so responsive and engaged, particularly so many years after creating the code. For that, I thank you.

I solved the problem of my image downloads stopping, and I hope this solution is helpful for other people.

What I found is that the database is timing out, because the image downloader is waiting for hosts that once had images that were posted in forum posts, that are not responding or no longer have the images on them.

What I do is to run the image downloader code manually from the vBulletin Administrative control panel, in the "Scheduled Task Manager" panel, hitting the "Run Now" button.

This forces your image downloader code to run manually (in between the 10-minute cron job period). The downloader code will run until it encounters an unresponsive host, and times out. There will be a database error (like I showed in the text I posted previously...you see the URL with the "charlesironside" text in it) that shows the host URL where the timeout occurred.

I note this in the database error, and then I add this host (not the entire URL, just the "www.xxxxx.com" portion or equivalent) to the "Ignore" list.

After adding the unresponsive host to the Ignore list, then I restart the image downloader manually (as above)...or wait for the cron job to start it again. It will automatically begin the downloading of the images, until it hits another unresponsive host.

I check on the downloading every couple of hours. If via my FTP program, I see that there have been no images downloaded for at least 30 minutes, then I run the code manually until I receive another database error. Then I add the offending host to the Ignore list, and start the process again.

So, what it comes down to, is hosts that are now off-line or otherwise unresponsive in posts made years ago.

This won't be an issue going forward (after all past posts' images are downloaded) because of the ongoing cron job, which will download external images from all future posts regularly -- before the hosts go off-line or otherwise become unresponsive.

I hope this makes sense, and I think it is probably the cause of many other peoples' problems with the image downloader abruptly stopping. The solution has worked for me and allowed me to "catch up" on the downloading for all of my past posts.

Regards,
Gerry

[y2ksw]

Originally Posted by gerryvz

Hi Giovanni,

Thank you for your responses. Rarely is a developer so responsive and engaged, particularly so many years after creating the code. For that, I thank you.
(...)
Regards,
Gerry

Hello Gerry,

I am an old style worker and it is my duty and pleasure to reply when I can

Regards,
Giovanni

[Afrika]

Originally Posted by y2ksw

Not without changing the code base.

But in reality, you may move the files directly and then change their path with a simple query in all of your posts. This is much more effective than importing and storing the images elsewhere.

UPDATE post SET pagetext=REPLACE(pagetext, 'old_url', 'new_url');

where old_url is the path as found in one in you posts now, and new_url the path as you wish to have as the new destination.

Keep in mind that the db connection may time out, but you can repeat as often as needed, since UPDATE is smart enough not to change the records which already have been changed. Best of course is to call mysql from the shell.

The problem is not to change the path, but there are too much pictures in the folder. Many of them are not used. The importer were the best possibility for that problem.

[y2ksw]

Originally Posted by Afrika

The problem is not to change the path, but there are too much pictures in the folder. Many of them are not used. The importer were the best possibility for that problem.

ok then:

• Move chunks of the pictures in subfolders and run the replacement query with regex for each chunk. However I am not familiar with the syntax, but there is https://dev.mysql.com/doc/refman/5.6/en/regexp.html

[Afrika]

Is there really no way to solve this with the importer? That would be an elegant way and the function to re-sort is already built.

[y2ksw]

Originally Posted by Afrika

Is there really no way to solve this with the importer? That would be an elegant way and the function to re-sort is already built.

You may change the code of the importer because it does not exclude the forum itself.

However, the most elegant function is to move the files and update the database with queries. It has a much smaller impact on a system than importing, namely the additional space it would eat up.

[Afrika]

OK thanks. What I have to change if the folder named picupload?