In addition, here is the error I received a few times relating to the database timing out. What changes can I make to the settings to avoid this?

Database error in vBulletin 4.2.4:

Invalid SQL:
INSERT IGNORE INTO vb_iei_img (
oldurl,
newurl
) VALUES (
'http://217.118.128.96/charlesironside/FMPro?-db=ecar2dta.fp5&key=41457&-img',
'https://www.500eboard.com/forums/clear.gif'
);

MySQL Error : MySQL server has gone away
Error Number : 2006
Request Date : Friday, November 17th 2017 @ 11:43:51 AM
Error Date : Friday, November 17th 2017 @ 11:44:52 AM
Script : http://www.500eboard.com/forums/admi...ron&cronid=171
Referrer : https://www.500eboard.com/forums/adm....php?do=modify
IP Address : 100.16.101.179
Username : gerryvz
Classname : vB_Database_MySQLi
MySQL Version :

Please try to disable the URL Cache. Then, all research moves from memory to database, a bit slower but less memory hungry.

The database server connection times out before the page or script finishes. This should never occur, but quite some hosting services "enforce" this policy in order to keep up with thousands of sites in a box. In a normal server, the connections time out once a day or so.

If this is your box, please make sure that the mysql server never drops connections, but also disallow permanent connections. The speed loss is won back by making less connections due to timed out pages and user refreshes.

Hi Giovanni,

Thank you for your responses. Rarely is a developer so responsive and engaged, particularly so many years after creating the code. For that, I thank you.

I solved the problem of my image downloads stopping, and I hope this solution is helpful for other people.

What I found is that the database is timing out, because the image downloader is waiting for hosts that once had images that were posted in forum posts, that are not responding or no longer have the images on them.

What I do is to run the image downloader code manually from the vBulletin Administrative control panel, in the "Scheduled Task Manager" panel, hitting the "Run Now" button.

This forces your image downloader code to run manually (in between the 10-minute cron job period). The downloader code will run until it encounters an unresponsive host, and times out. There will be a database error (like I showed in the text I posted previously...you see the URL with the "charlesironside" text in it) that shows the host URL where the timeout occurred.

I note this in the database error, and then I add this host (not the entire URL, just the "www.xxxxx.com" portion or equivalent) to the "Ignore" list.

After adding the unresponsive host to the Ignore list, then I restart the image downloader manually (as above)...or wait for the cron job to start it again. It will automatically begin the downloading of the images, until it hits another unresponsive host.

I check on the downloading every couple of hours. If via my FTP program, I see that there have been no images downloaded for at least 30 minutes, then I run the code manually until I receive another database error. Then I add the offending host to the Ignore list, and start the process again.

So, what it comes down to, is hosts that are now off-line or otherwise unresponsive in posts made years ago.

This won't be an issue going forward (after all past posts' images are downloaded) because of the ongoing cron job, which will download external images from all future posts regularly -- before the hosts go off-line or otherwise become unresponsive.

I hope this makes sense, and I think it is probably the cause of many other peoples' problems with the image downloader abruptly stopping. The solution has worked for me and allowed me to "catch up" on the downloading for all of my past posts.

Regards,
Gerry

Hello Gerry,

I am an old style worker and it is my duty and pleasure to reply when I can :)

Regards,
Giovanni

The problem is not to change the path, but there are too much pictures in the folder. Many of them are not used. The importer were the best possibility for that problem.

ok then:

• Move chunks of the pictures in subfolders and run the replacement query with regex for each chunk. However I am not familiar with the syntax, but there is https://dev.mysql.com/doc/refman/5.6/en/regexp.html

Is there really no way to solve this with the importer? That would be an elegant way and the function to re-sort is already built.

You may change the code of the importer because it does not exclude the forum itself.

However, the most elegant function is to move the files and update the database with queries. It has a much smaller impact on a system than importing, namely the additional space it would eat up.

OK thanks. What I have to change if the folder named picupload?

Depending on what you have decided to do, you have to change the plugin code or move the files.

Code changes: The code is commented and uses self-explanatory variables, thus you should be able to find easily the place where to skip the forum path test and comment that part out or change it to your needs. Alternatively, you may hire somebody (not me) who is going to make the changes.

Move files manually: use shell commands and phpmyadmin for the queries ;)

I imagine this would work fine with vb3 & vb5?

Sorry, no. There was a 3.x version and there is no 5.x version and probably never will be.

Thanks, is there a way to make it only import http images (Non SSL images)? That is, skip any images with https in it and only download the "mixed content" http images.

I noticed that sometimes this plugin corrupts the filenames. Any fix for that?

please release 3.8.11 for vbulletin.

In the settings for the plugin (Settings -> Options -> vB4 Import External Images), put "https://" (without quotes) on a new line in the "Ignore" section, like so:

Attachment 157170

You may consider downloading everything though, if you have the space/bandwidth to support it, because otherwise, external hosts could get rid of their images - either due to age, or some ulterior motive, and by surprise, like Photobucket.

Hi and thanks for your efforts.

I just have a question, I noticed that the imported folder has to be chmod to 777, however all the imported files are automatically chmod to 644.

In this case, if a user posts an "infected" image that contains a php script or malicious code, are they blocked by chmod 644 or is there still a chance that the code gets executed when the hacker opens the url mysite.com/images/imported/infectedfile.png ?

Thanks

CHMOD/CHOWN are server based permissions and have no bearing to the end user if the file is served to them. What you need is to have an addon that uses FProt or ClamAV to scan attachments and imported files. I know there was Fractalizer for VB 3 and there was a VB4 variant but I think they were only for attachments. This is something that would need to be coded from scratch.

OK thanks for your fast reply, that means that this plugin is pretty insecure? In this case is there a way to put an .htaccess in the imported folder that would deny anyone from running php scripts?

The plugin in itself is not insecure as I do not believe that the intention of scanning files for malicious code was ever its intent. That being said you can then say vBulletin is not secure in that the way it allows you to have attachments and upload avatars. There is a known exploit:

The following issues exist in vBulletin itself, reported by us to vBulletin support over 60 days ago(years now). Since vBulletin has not patched or disclosed the issues since that time, we now do so here:
An image decompression bomb vulnerability exists when vBulletin Options > Message Attachment Options > Resize Images = Yes. Disable it to protect your site.
An image decompression bomb vulnerability exists when allowing user uploads for avatars and profile pictures. To protect your site, change your forum's permissions so that users cannot upload custom avatars or profile pics.
An image decompression bomb vulnerability exists when using ImageMagick for images and allowing uploads. Currently known issues are for PDFs and TIFFs; however, because the filename of the incoming upload is not trustworthy, removing entries from the Attachment Manager or changing Attachment Permissions are not viable options. The following mitigation options exist:
Change vBulletin Options > Image Settings > Image Processing Library = GD, OR
Change your forum's permissions so that no users can upload anything.

In an attempt to figure the best way to resolve this sort of thing would be to update the Fractalizer Plugin to work with VB 4: https://vborg.vbsupport.ru/showthread.php?t=187482

Any fix for PHP 7.1+

PHP Fatal error: [] operator not supported for strings on line 324 in /home/........./public_html/forums/includes/cron/iei_cron.php

Edit
to
Or better to
so the following code line numbers are the same. I expect there are more than one fatal errors in.


Edit:
former edit of line 342 was stupid. Deleted

Thank you for the quick reply :)
File edits made. Will check the log file latter on tonight and see if anything pops back up.

Still getting (1) error with this...

NOTICE: PHP message: PHP Fatal error: syntax error, unexpected ';', expecting ']' on line 334 in /home/........./public_html/forums/includes/cron/iei_cron.php"

This did not work...

Trying the first notation...

I'm very unhappy with PHP 7, which converts the untyped language in a typed one. Having said this, you may test

if (isset($iei_ignore)) ...

in order to see if the variable was actually set, or better, use a different variable to read the value, and use

$iei_ignore = array();

beforehand. E.g.:

$iei_ignore = array();
$x_ignore = explode("\r\n", $input_ignore);
if (!is_array($x_ignore))
{
$iei_ignore[] = $bburl;
}
else
{
$iei_ignore = $x_ignore;
}

Then you made a typing error. "syntax error, unexpected ';', expecting ']'" is not a php7 error or so. And you ar sure line 334? the qoted lines are in 324. Your quoted code can't be your inserted code.

But of course you can use the other version or from y2ksw (but the if conditions there is wrong I think)

$image_tags = array('img');

As far as I know, implicit arrays are not allowed for a long time by now, but explicit arrays like above, must work.

Is there anyway to get this to support a URL such as:

If I try it comes back as invalid as I suspect it is looking for an image extension such as .jpg, .png, etc.

I was going to suggest you strip off everything after the .jpg extension so it would just be this:

but that URL gives an error: Bad URL timestamp

Maybe because it's a CDN, possibly a private CDN?

Not sure, but it should be able to pull down the image or at least I was hoping it would...

Not if the server is misconfigured or the link isn't accessible to the public.

I can enter the entire URL and the image is displayed from multiple locations. But as you pointed out it is more than likely with the code after the named image.

Ah yes. So it does.

I know that some online news services use similar formats to discourage or prevent image scraping. You may need to just accept that such images won't be imported. Alternatively, you can download the image from your browser and upload it from your local disk as an embedded attachment.

Because of what my site is/does, it is not something that I can do. I am just trying to supply the tools. I'll probably just have to have a developer look at it and see if it can be coded to accept URLs of this nature. Thanx for the replies :)

These URL's normally don't work, because they are usually a sub-set of a larger service.

It would eventually work, if the part after, including ? is removed. However it looks like a temporary link anyway.

Hello Giovanni..

Hope you have been well. It's been a long time and I'm not sure that you are still active here.

I've been using your Beta 1.0.6 for several years with vBulletin 4.1.3pl1 and it’s been work great!

I’ve had to update to bulletin 4.25 now and I’m not sure what’s gone wrong, but the attachment files read by 1.0.6 into directory /images/imported are now showing with only a “?” where the picture should be?

When I check the link for any pic, it’s showing as an example.

/attachment.php?attachmentid=121951&d=1639421597

…. when it should be linking to a pic captured by your mod and stored within /images/imported?

I have the directory in your "Vb Import External Images" options page set to /images/imported.

I’m not sure what else I need to do to get into work with vBulletin 4.2.5 as it always has with 4.13pl1?

Can you help at all?

Thank you…

Regards,
Doug

Hello Doug

In fact I have more or less moved away from web and forums, but I try to keep up - you never know.

I'm not sure what might have happened, but I guess, something at the server side has changed ... or the page encoding, style, PHP version, fopen permissions ... or the script was copied incomplete or was modified, or the database has a different encoding. Have you tried to change the setting and and change it again, in order to store it as a "new" setting?

Seeking a wee bit of advice of this mod. I'm in the process of rebuilding a vbforum from an old website that is now defunct. The old website ran this mod as an addon and it proved very useful. However, the restored database that I'm preping is currently running on a local machine on my home network so the URLs have all changed from FQDN to an IP address.

I've found a table in the database called IEI_IMG and it seems to me that all I need to do here is do a search and replace on the NewURL field and replace the sitename.com with IP address of the local machine on which it is running. Can anyone confirm that this is all I need to do to point the URL's for all the imported images to a new path on my local machine? Or will I have to single-step through every thread and replace every single URL?

I suggest to add your domain to the hosts file and access your local site as if it was remote.

Thanks for your reply Giovanni, I got it sorted in the end by doing a string replace in the database. The IEI_IMG table was just a simple record of what had been imported and from where, this was invaluable in helping me rebuild missing content so good call in including it in the package!

Is there any chance that this mod can be ported to VB5x, it's absolutely invaluable on my site!