Import External Images

[Dave]

@y2ksw I think I know the cause, I've made the following changes for someone and now it's working again:

In the cURL part in the iei_get_contents function, I made it so that any SSL errors are ignored (for HTTPS sites). This can be done by setting CURLOPT_SSL_VERIFYHOST and CURLOPT_SSL_VERIFYPEER to 0. I also added a check there so that it returns false when the status code is bigger than 400.

Finally, the iei_normalize_path function adds a slash in front of the path which does not seem to work on all servers. I removed the first slash in the path that's created by that function and that fixed the whole thing.

[BadgerDog]

@y2ksw I think I know the cause, I've made the following changes for someone and now it's working again:

In the cURL part in the iei_get_contents function, I made it so that any SSL errors are ignored (for HTTPS sites). This can be done by setting CURLOPT_SSL_VERIFYHOST and CURLOPT_SSL_VERIFYPEER to 0. I also added a check there so that it returns false when the status code is bigger than 400.

Finally, the iei_normalize_path function adds a slash in front of the path which does not seem to work on all servers. I removed the first slash of the path that's created and that fixed the whole thing.


Interesting Dave .... thanks .. :)

We import pics from all kinds of SSL sites and have so for years with this mod, but I don't recall us ever having to modify CURL options to do so.. and y2ksw's mod hasn't crashed, or failed to import a pic in years.

Are we missing something, or are we just lucky?

Regards,
Doug

[Dave]

Interesting Dave .... thanks .. :)

We import pics from all kinds of SSL sites and have so for years with this mod, but I don't recall us ever having to modify CURL options to do so.. and y2ksw's mod hasn't crashed, or failed to import a pic in years.

Are we missing something, or are we just lucky?

Regards,
Doug

I assume you're just lucky. :)

[Rideharder]

Hello guys will this work on Version 4.2.5 or would you guys recommend something else for that 4.2.5? Thanks!!!

[y2ksw]

Hello guys will this work on Version 4.2.5 or would you guys recommend something else for that 4.2.5? Thanks!!!
It should work, but I can't promise ... please try and let us know :)

[Kane@airrifle]

Works in 4.2.5 PHP 7.1 for me.

[Paul M]

Hello guys will this work on Version 4.2.5 or would you guys recommend something else for that 4.2.5? Thanks!!!
I dont see why not, I have it (or at least a modified version of it) working on 3.8.11 & PHP 7.1, which is basically the same as far as this code is concerned.

(** The modifications relate to the information stored, I like to save the postid and a timestamp).

[Rideharder]

Awesome, thank you so much. I remember I stop doing the Vbulletin because it was a pain in the ass to keep up with all the version updates on the modules. You guys are pretty awesome to have around for the community and thanks!

[andy626]

Hi folks,

Some members pointed me to this thread, great mod!

Now that Photobucket has stopped 3rd party hosting, years of threads on my forum now have the below redirected image.

Has anyone tried this mod since PB have done this?

Any luck?

I'm concerned it will try and pull all of the PB photos but will pull the redirected image code instead or just blanks, any ideas/help greatly appreciated!

https://vborg.vbsupport.ru/

[Kane@airrifle]

Hi folks,

Some members pointed me to this thread, great mod!

Now that Photobucket has stopped 3rd party hosting, years of threads on my forum now have the below redirected image.

Has anyone tried this mod since PB have done this?

Any luck?

I'm concerned it will try and pull all of the PB photos but will pull the redirected image code instead or just blanks, any ideas/help greatly appreciated!

https://vborg.vbsupport.ru/

Bad news, in testing I have found that the image is replace with the PB shakedown thumbnail

[andy626]

Bad news, in testing I have found that the image is replace with the PB shakedown thumbnail Oh crap, disaster!

But a million thank you's for testing this for me!

I think I still need this mod for Tapatalk and Forum Runner, etc.

Does it completely ruin the thread if I was to somehow find a script to find and replace photobucket.com/member/image123.jpg to myforum.com/member/image123 etc.

i.e: is the localised image link still the correct name/folder format of the original image or is the new image name the 3rd party thumbnail image link..... <--- Not even sure that will make sense to you :confused:

Appreciate your help though!

[andy626]

This is a bit of a disaster for me and I'm sure hundreds of forums out there surely?

What would be good if you can somehow get members to download their albums and photos off PB.

Then you upload it your server and somehow (???) run an attachment URL find and replace PB.com/member/ to yourforum.com/member/

Once that's been done, you don't need PB and then run this script here for the rest of the 3rd party hosting.

I never knew how vulnerable we are as forum owners when companies can suddenly change their ways and charge $399 per year to show your photos on 3rd party, crazy!

[Kane@airrifle]

This product will only download and replace images in the post table that are between the [img] [ /img] tags so I don't t think it can help you with member profile pics and member albums.

The localised path will be designated by you on setup, you have a choice whether or not to keep the original filenames or to have the mod change them to a numeric value name.

It is a disaster and I am sure there are millions of websites affected. I am pretty glad I installed this mod shortly after Imageshack pulled the same stunt that Photobucket is doing now.

[andy626]

Thanks again kane, I'm running this mod now excluding PB and its working great!

Can't believe I didn't have this on my forum earlier!!!

I'm asking for any help to somehow run an attachment find and replace URL script in this thread below if anyone can help I would be extremely grateful...

https://vborg.vbsupport.ru/showthread.php?p=2588046

[The_Hawk]

I'm in the same boat as everyone else since the photobucket issues. Sadly this mod has been on my todo list for almost a year and I never quite got there :(

It's now installed and running nicely (excluding photobucket at the moment), although with a shade under 1.1 million posts to trawl through it's going to take a while. I'm on a dedicated box so have pushed it up to 1000 posts per run and it seems to be coping (although I had to disable URL caching as it ran out of memory very quickly).

On the photobucket side there are a range of plugin for Chrome which claim to get around the issue, some are marked as dead already but at lease one is working for me.

While I'm sure that Photobucket will be killing these sorts of work around off as quickly as they can, I do wonder if this sort of approach could be built into this mod, even if it's only short lived, giving us all a chance to pull as many images as we can, while we can.

I have no idea what this would take, but I'd happily put in some $$ for the time it takes to get this up and running.

https://chrome.google.com/webstore/detail/photobucket-hotlink-fix/kegnjbncdcliihbemealioapbifiaedg

[The_Hawk]

Actually I spoke too soon, the cron job seems to have stopped running for no apparent reason. Even trying to tell it to "run now" seems to not work.

I did hit the "Allowed memory size exhausted" error, but turned off caching... then cleared the iei_img table as well. Still no luck.

I've set the number of posts back down to 10, re-imported the mod, checked the permissions on the folders, rebooted the server... still no clue.

Other tasks still appear to be running automatically or using the "run now" button (which then shows "done"), just not this one.

I can't seem to find any error message anywhere that would point to why this is failing. Any suggestions are welcome.

[The_Hawk]

So I've had a few messages back and forth with y2ksw, which has been brilliant.

The latest advice was the PHP might be silently crashing and wanted more memory. I've been slowly upping the max_memory (as I didn't want to go unlimited just yet) and it's running again. I'm somewhere north of 40,000 images checked so far (excluding photobucket) and I'm currently running at only about 40% of links being found. To be fair this is
2005 - 2011 so it's less of a surprise but really does ram home the need to run a script like this one if possible.



We had this issue in the past with very large images (like 10.000x20.000 pixel) and PHP silently crashing upon insufficient memory. If you are on a dedicated box, you may try to assign "-1" (=any) to max memory. Else, you may try to sign the first unscanned post as scanned to skip over the post which creates this problem.

[Kane@airrifle]

A work around for the PB issue. edit /includes/cron/iei_cron; after line 268 add:

if (is_numeric(stripos($url, 'photobucket')))
{
curl_setopt($ch, CURLOPT_REFERER, 'http://www.photobucket.com/');
}

I have tested it on a few known blocked PB images and it imports them just fine.

*not my code, was posted elsewhere by "IA1"

[fwulfers]

Thanks so much for this plugin and the code update for Photobucket. I installed in on our forum (vB 4.2.5), made the change to the code and it downloads the PB images just fine. Many tutorials and other important threads will now be useful again.

[y2ksw]

... and most of all, you never lose them again :)

[BadgerDog]

A work around for the PB issue. edit /includes/cron/iei_cron; after line 268 add:

if (is_numeric(stripos($url, 'photobucket')))
{
curl_setopt($ch, CURLOPT_REFERER, 'http://www.photobucket.com/');
}

I have tested it on a few known blocked PB images and it imports them just fine.

*not my code, was posted elsewhere by "IA1"

For my clarity ... :)

I donated via PayPal and have run y2ksw's for many many years. It's been fantastic and one of the most popular add-ons our members comment on.

My question ... since it's been running for so long and we already have thousands of PB pics imported onto our server, am I correct in assuming that this code change permits members to continue IMG'ing PB pics and they will still import?

In other words, this code fix works around PB's block and returns us to the status quo of what it used to be?

Thanks for any feedback?

Regards,
Doug

Edit: I don't have a line counter in my editor, so what does the code that appears before this look like so I'm sure to put this in the right place, if I do i... thanks ..

[Kane@airrifle]

Doug, the PB fix will work until such time as they find a means to obfusticate it again.

In the iei_cron.php file look for the following code and add the PB code directly after it:

@curl_setopt($ch, CURLOPT_REFERER, IEI_BBURL);

In the long run it would assist you greatly if you used something like Notepad++ to edit your files with: https://notepad-plus-plus.org/

[BadgerDog]

Doug, the PB fix will work until such time as they find a means to obfusticate it again.

In the iei_cron.php file look for the following code and add the PB code directly after it:

@curl_setopt($ch, CURLOPT_REFERER, IEI_BBURL);

In the long run it would assist you greatly if you used something like Notepad++ to edit your files with: https://notepad-plus-plus.org/

Thank you partner.... :)

Appreciate the quick response .. :up:

Regards,
Doug

ps: I'm on a MacBook Pro OSX operating system ..

pss: Ok, I've now inserted it for testing and it looks like this... seem like the right place?

if(USE_CURL)
{
$ch = @curl_init($url);
@curl_setopt($ch, CURLOPT_BINARYTRANSFER, 1);
@curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
@curl_setopt($ch, CURLOPT_FAILONERROR, 1);
@curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
@curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, IEI_CONNECTTIMEOUT);
@curl_setopt($ch, CURLOPT_TIMEOUT, IEI_CURL_TIMEOUT);
@curl_setopt($ch, CURLOPT_REFERER, IEI_BBURL);
if (is_numeric(stripos($url, 'photobucket')))
{
curl_setopt($ch, CURLOPT_REFERER, 'http://www.photobucket.com/');
}
//@curl_setopt($ch, CURLOPT_USERAGENT, IEI_FORUMDOMAIN);
$contents = @curl_exec($ch);
@curl_close($ch);
}

psss: Well I ran it anyway and it didn't crash (see pic), so I guess I have to wait now until I can get someone to try a PB pic, as I don't use their service...

[y2ksw]

For my clarity ... :)

I donated via PayPal and have run y2ksw's for many many years. It's been fantastic and one of the most popular add-ons our members comment on.

My question ... since it's been running for so long and we already have thousands of PB pics imported onto our server, am I correct in assuming that this code change permits members to continue IMG'ing PB pics and they will still import?

In other words, this code fix works around PB's block and returns us to the status quo of what it used to be?

Thanks for any feedback?

Regards,
Doug

Edit: I don't have a line counter in my editor, so what does the code that appears before this look like so I'm sure to put this in the right place, if I do i... thanks ..
Sooner or later they will elaborate more and will not accept only their base url as a referrer. Until then, this code modification should work.

[Kane@airrifle]

ps: I'm on a MacBook Pro OSX operating system ..

Try something like Atom then: https://atom.io/

[BadgerDog]

Try something like Atom then: https://atom.io/

Thanks again .. :up:

Regards,
Doug

[The_Hawk]

A work around for the PB issue. edit /includes/cron/iei_cron; after line 268 add:

if (is_numeric(stripos($url, 'photobucket')))
{
curl_setopt($ch, CURLOPT_REFERER, 'http://www.photobucket.com/');
}

I have tested it on a few known blocked PB images and it imports them just fine.

*not my code, was posted elsewhere by "IA1"

You Sir, are a legend! (and so is IA1 for this remarkably simple fix!!)

It's ticking away nicely pulling images as we speak! I'm sitting on the edge of my seat watching it and testing images as they pull down to make sure they are continuing to work... LOVE IT!!

Try something like Atom then: https://atom.io/

I like BBEdit on the Mac and Notepad++ on Windows. My old job used to use EM Editor which was great for very large files (1M+ lines files dumped from finance systems), but Notepad++ works fine for my coding needs.

BBEdit:
http://www.barebones.com/

EM Editor:
https://www.emeditor.com/

[BadgerDog]

A work around for the PB issue. edit /includes/cron/iei_cron; after line 268 add:

if (is_numeric(stripos($url, 'photobucket')))
{
curl_setopt($ch, CURLOPT_REFERER, 'http://www.photobucket.com/');
}

I have tested it on a few known blocked PB images and it imports them just fine.

*not my code, was posted elsewhere by "IA1"

Just a follow up and again, a huge thank you for this quick fix (for now)... :up:

It's working beautifully and being a research site for over 29,000 retired veterans, is extremely valuable to our members... much appreciated...

We've been using y2ksw's outstanding add-on for over 7 years and the accumulated amount of important Photo Bucket research pics that have been preserved (saved) by his mod over the years is measured in the 10's of thousands.

Your temporary fix allows that to continue with new pics, at least until Photo Bucket catches up with it. :)

Regards,
Doug

[rkrenicki]

I think I am missing something real simple here.. but I am not finding a log entry for this mod. Does this not have logging functions, or am I missing where to enable it?

EDIT: Nevermind, i found it and it is logging now.

[Kane@airrifle]

It's not set to log the cron action by default but it does write the imports (old_url - new_url) to a DB table called "iei_img"

[M4T VW]

Doesn't seem to be working with photobucket again? It just downloads a load of "no external hosting allowed" images.

[y2ksw]

Doesn't seem to be working with photobucket again? It just downloads a load of "no external hosting allowed" images.
Yeah, they might keep an eye on it or eventually apply a firewall rule to keep people from downloading the images.

[y2ksw]

Yeah, they might keep an eye on it or eventually apply a firewall rule to keep people from downloading the images.
Maybe you better give up on them. I tried to get to the point where an image shows up, but no success.

It seems they have disabled public images and were "sold" to an unnamed third party which now made it an advertising hog. It's a pity how deep certain services may fall ;)

[Kane@airrifle]

I did a PB import today for somebody and got 96/100 imported.

[unitedpunjab]

Is it possible to 'reverse' the ignore option ?
So that only the url specified can be cached.

I will try to toggle some true/false but it will be helpful if someone has already done this.

[Gizmo99]

just found that if you add ~original after the .jpg the image of the photobucket [IMG] link and the image appears :)

can or will this script do this ?

Giz

[The_Hawk]

The fix still appears to be working for me with images imported as late as yesterday without issues. I do note that I have *very* few images coming through from photobucket as people have moved onto other things. It could be that those last couple are paid accounts and that's why it's working... For one main user it probably is, for the other couple it's unlikely.

Again, brilliant little fix that the community is loving.

[VCCN]

Very good plugin, works exellent, thank you :)
One question; is it possible to have the plugin to fetche photobucket-links only?

[VCCN]

Update: seems like PB has blocked this now. Alot of PB-pictures is not changed according to log, when i check the actual post, and add ~ behind the .jpg the picture works fine :-(

Edit: checked the table and many PB-pictures is downloaded, and some is not. hmm

[y2ksw]

Very good plugin, works exellent, thank you :)
One question; is it possible to have the plugin to fetche photobucket-links only?
No. The idea is to download all images.

[y2ksw]

Update: seems like PB has blocked this now. Alot of PB-pictures is not changed according to log, when i check the actual post, and add ~ behind the .jpg the picture works fine :-(

Edit: checked the table and many PB-pictures is downloaded, and some is not. hmm
If some images do not download, it may be that some PB accounts are simply moved elsewhere.

[VCCN]

If some images do not download, it may be that some PB accounts are simply moved elsewhere.

Don't think so.
If i check a post that is "Not changed" according to log - i find the "you must upgrade your PB" picture. If i copy the URL and open it up in a browser, picture is showing up.
Not sure why, maybe because of the time out?
Anyway, how can i rerun this plugin on the whole forum? Let's say the plugin has scanned and downloaded all pictures and i want to start over - how do i do it?

[y2ksw]

Don't think so.
If i check a post that is "Not changed" according to log - i find the "you must upgrade your PB" picture. If i copy the URL and open it up in a browser, picture is showing up.
Not sure why, maybe because of the time out?
Anyway, how can i rerun this plugin on the whole forum? Let's say the plugin has scanned and downloaded all pictures and i want to start over - how do i do it?
The point is, in any case, a server is not a browser. When simple attempts to fake a browser fail, then you have two options: you develop a software which handles that specific case and repair it every time something changes, or you say: impossible.

I do not have time to watch PB developers, so to me it's the second option :)

[BGObsession]

So I've had a few messages back and forth with y2ksw, which has been brilliant.

The latest advice was the PHP might be silently crashing and wanted more memory. I've been slowly upping the max_memory (as I didn't want to go unlimited just yet) and it's running again. I'm somewhere north of 40,000 images checked so far (excluding photobucket) and I'm currently running at only about 40% of links being found. To be fair this is
2005 - 2011 so it's less of a surprise but really does ram home the need to run a script like this one if possible.

I believe I am experiencing this as well. I installed this mod in order to overcome the 'mixed content' warnings and so far it is not working for me. I see a few images have been added to the 'imported' image folder, but just a few, and I also get the 'Done' (and only 'Done') message when I manually run the cron.job.

When you say you are 'upping the max_memory' (or to set it to 'unlimited') - where are those adjustments done - is that within the vB admin panel somewhere, or are you talking about a server setting. How can I adjust this in order to get the cron job to completely run. And I'm assuming if this is working 100% correctly, that I should see the mixed content warnings disappear immediately - is that right?

[y2ksw]

I believe I am experiencing this as well. I installed this mod in order to overcome the 'mixed content' warnings and so far it is not working for me. I see a few images have been added to the 'imported' image folder, but just a few, and I also get the 'Done' (and only 'Done') message when I manually run the cron.job.

When you say you are 'upping the max_memory' (or to set it to 'unlimited') - where are those adjustments done - is that within the vB admin panel somewhere, or are you talking about a server setting. How can I adjust this in order to get the cron job to completely run. And I'm assuming if this is working 100% correctly, that I should see the mixed content warnings disappear immediately - is that right?
As far as I know the only place to lay hands on, is the php.ini file and physical memory.
# cat /etc/php5/apache2/php.ini | grep memory_limit

[Emad ELsayed]

my dear
Does not work on 4.2.5
php 5.6.31
mysql 5.6.37
any help ..
Best Regards

[y2ksw]

my dear
Does not work on 4.2.5
php 5.6.31
mysql 5.6.37
any help ..
Best Regards
for me it works

[Afrika]

Is it possible to import images from a specific path from the own forum? The own forum is not imported by default

[y2ksw]

Is it possible to import images from a specific path from the own forum? The own forum is not imported by default
Not without changing the code base.

But in reality, you may move the files directly and then change their path with a simple query in all of your posts. This is much more effective than importing and storing the images elsewhere.

UPDATE post SET pagetext=REPLACE(pagetext, 'old_url', 'new_url');

where old_url is the path as found in one in you posts now, and new_url the path as you wish to have as the new destination.

Keep in mind that the db connection may time out, but you can repeat as often as needed, since UPDATE is smart enough not to change the records which already have been changed. Best of course is to call mysql from the shell.

[gerryvz]

Hi Giovanni,

Great mod, and thank you so much for creating it. I am running 4.2.4 and installed the mod yesterday. I have around 7,300 users and 150,000 total posts, with a relatively small number of overall posts that would require images to be downloaded (I would say the majority of images on my forum are directly attached and residing already in the file system).

Server - Linux + Apache
vbulletin - 4.2.4
php - 5.6.31
php max post size - 64MB
php max upload size - 64MB
php memory limit - 256MB
mySQL - 5.5.7
mySQL packet size - 10MB

The mod started downloading and it got through about 4 of the 9 years of posts on my forum. Now it has frozen since last night. I turned on the logging this morning and only came across this single entry in the log since I turned it on, which is attached. The offending post has some images attached to it, using the external host home.btconnect.com

The images seem to be small in size and pixels, and I downloaded them manually and attached them to the post to see if I could have the mod continue downloading other posts. So far, I haven't been able to get it started again.

Attached is the error, as well as my settings screens.

Also, I have now received two times the mySQL error as attached, which I read appears to be the database server "timing out."

The mod is definitely enabled and running, download folder and sub-folders are 777 permissions, etc.

Does it look to you that there is an image that is too large, or do you have other ideas as to what is causing the mod to stop indexing and downloading?

EDIT: A new post that was made by one of my users today was indexed and the external images were replaced, so I know the code is working. However, it seems stalled and won't continue indexing beyond a certain point in the past. It must be hanging up on something that is keeping it from continuing past indexing. Please advise :)

SECOND EDIT: I emptied the table in the database (it has a few more than 3,000 records in it) and the code executed again, but then stopped, and it had seven new records in the database. I emptied the table again and it sent through one more cycle of downloading a few more images, and putting seven new records in the database again.

So, something related to the database and/or php memory seems to be preventing the code from executing correctly.

Any thoughts?

Thanks
Gerry






[gerryvz]

In addition, here is the error I received a few times relating to the database timing out. What changes can I make to the settings to avoid this?

Database error in vBulletin 4.2.4:

Invalid SQL:
INSERT IGNORE INTO vb_iei_img (
oldurl,
newurl
) VALUES (
'http://217.118.128.96/charlesironside/FMPro?-db=ecar2dta.fp5&key=41457&-img',
'https://www.500eboard.com/forums/clear.gif'
);

MySQL Error : MySQL server has gone away
Error Number : 2006
Request Date : Friday, November 17th 2017 @ 11:43:51 AM
Error Date : Friday, November 17th 2017 @ 11:44:52 AM
Script : http://www.500eboard.com/forums/admincp/cronadmin.php?do=runcron&cronid=171
Referrer : https://www.500eboard.com/forums/admincp/cronadmin.php?do=modify
IP Address : 100.16.101.179
Username : gerryvz
Classname : vB_Database_MySQLi
MySQL Version :

[y2ksw]

Hi Giovanni,

Great mod, and thank you so much for creating it. I am running 4.2.4 and installed the mod yesterday. I have around 7,300 users and 150,000 total posts, with a relatively small number of overall posts that would require images to be downloaded (I would say the majority of images on my forum are directly attached and residing already in the file system).

Server - Linux + Apache
vbulletin - 4.2.4
php - 5.6.31
php max post size - 64MB
php max upload size - 64MB
php memory limit - 256MB
mySQL - 5.5.7
mySQL packet size - 10MB

The mod started downloading and it got through about 4 of the 9 years of posts on my forum. Now it has frozen since last night. I turned on the logging this morning and only came across this single entry in the log since I turned it on, which is attached. The offending post has some images attached to it, using the external host home.btconnect.com

The images seem to be small in size and pixels, and I downloaded them manually and attached them to the post to see if I could have the mod continue downloading other posts. So far, I haven't been able to get it started again.

Attached is the error, as well as my settings screens.

Also, I have now received two times the mySQL error as attached, which I read appears to be the database server "timing out."

The mod is definitely enabled and running, download folder and sub-folders are 777 permissions, etc.

Does it look to you that there is an image that is too large, or do you have other ideas as to what is causing the mod to stop indexing and downloading?

EDIT: A new post that was made by one of my users today was indexed and the external images were replaced, so I know the code is working. However, it seems stalled and won't continue indexing beyond a certain point in the past. It must be hanging up on something that is keeping it from continuing past indexing. Please advise :)

SECOND EDIT: I emptied the table in the database (it has a few more than 3,000 records in it) and the code executed again, but then stopped, and it had seven new records in the database. I emptied the table again and it sent through one more cycle of downloading a few more images, and putting seven new records in the database again.

So, something related to the database and/or php memory seems to be preventing the code from executing correctly.

Any thoughts?

Thanks
Gerry





Please try to disable the URL Cache. Then, all research moves from memory to database, a bit slower but less memory hungry.

[y2ksw]

In addition, here is the error I received a few times relating to the database timing out. What changes can I make to the settings to avoid this?

Database error in vBulletin 4.2.4:

Invalid SQL:
INSERT IGNORE INTO vb_iei_img (
oldurl,
newurl
) VALUES (
'http://217.118.128.96/charlesironside/FMPro?-db=ecar2dta.fp5&key=41457&-img',
'https://www.500eboard.com/forums/clear.gif'
);

MySQL Error : MySQL server has gone away
Error Number : 2006
Request Date : Friday, November 17th 2017 @ 11:43:51 AM
Error Date : Friday, November 17th 2017 @ 11:44:52 AM
Script : http://www.500eboard.com/forums/admincp/cronadmin.php?do=runcron&cronid=171
Referrer : https://www.500eboard.com/forums/admincp/cronadmin.php?do=modify
IP Address : 100.16.101.179
Username : gerryvz
Classname : vB_Database_MySQLi
MySQL Version :
The database server connection times out before the page or script finishes. This should never occur, but quite some hosting services "enforce" this policy in order to keep up with thousands of sites in a box. In a normal server, the connections time out once a day or so.

If this is your box, please make sure that the mysql server never drops connections, but also disallow permanent connections. The speed loss is won back by making less connections due to timed out pages and user refreshes.

[gerryvz]

Hi Giovanni,

Thank you for your responses. Rarely is a developer so responsive and engaged, particularly so many years after creating the code. For that, I thank you.

I solved the problem of my image downloads stopping, and I hope this solution is helpful for other people.

What I found is that the database is timing out, because the image downloader is waiting for hosts that once had images that were posted in forum posts, that are not responding or no longer have the images on them.

What I do is to run the image downloader code manually from the vBulletin Administrative control panel, in the "Scheduled Task Manager" panel, hitting the "Run Now" button.

This forces your image downloader code to run manually (in between the 10-minute cron job period). The downloader code will run until it encounters an unresponsive host, and times out. There will be a database error (like I showed in the text I posted previously...you see the URL with the "charlesironside" text in it) that shows the host URL where the timeout occurred.

I note this in the database error, and then I add this host (not the entire URL, just the "www.xxxxx.com" portion or equivalent) to the "Ignore" list.

After adding the unresponsive host to the Ignore list, then I restart the image downloader manually (as above)...or wait for the cron job to start it again. It will automatically begin the downloading of the images, until it hits another unresponsive host.

I check on the downloading every couple of hours. If via my FTP program, I see that there have been no images downloaded for at least 30 minutes, then I run the code manually until I receive another database error. Then I add the offending host to the Ignore list, and start the process again.

So, what it comes down to, is hosts that are now off-line or otherwise unresponsive in posts made years ago.

This won't be an issue going forward (after all past posts' images are downloaded) because of the ongoing cron job, which will download external images from all future posts regularly -- before the hosts go off-line or otherwise become unresponsive.

I hope this makes sense, and I think it is probably the cause of many other peoples' problems with the image downloader abruptly stopping. The solution has worked for me and allowed me to "catch up" on the downloading for all of my past posts.

Regards,
Gerry

[y2ksw]

Hi Giovanni,

Thank you for your responses. Rarely is a developer so responsive and engaged, particularly so many years after creating the code. For that, I thank you.
(...)
Regards,
Gerry
Hello Gerry,

I am an old style worker and it is my duty and pleasure to reply when I can :)

Regards,
Giovanni

[Afrika]

Not without changing the code base.

But in reality, you may move the files directly and then change their path with a simple query in all of your posts. This is much more effective than importing and storing the images elsewhere.

UPDATE post SET pagetext=REPLACE(pagetext, 'old_url', 'new_url');

where old_url is the path as found in one in you posts now, and new_url the path as you wish to have as the new destination.

Keep in mind that the db connection may time out, but you can repeat as often as needed, since UPDATE is smart enough not to change the records which already have been changed. Best of course is to call mysql from the shell.
The problem is not to change the path, but there are too much pictures in the folder. Many of them are not used. The importer were the best possibility for that problem.

[y2ksw]

The problem is not to change the path, but there are too much pictures in the folder. Many of them are not used. The importer were the best possibility for that problem.
ok then:


Move chunks of the pictures in subfolders and run the replacement query with regex for each chunk. However I am not familiar with the syntax, but there is https://dev.mysql.com/doc/refman/5.6/en/regexp.html

[Afrika]

Is there really no way to solve this with the importer? That would be an elegant way and the function to re-sort is already built.

[y2ksw]

Is there really no way to solve this with the importer? That would be an elegant way and the function to re-sort is already built.
You may change the code of the importer because it does not exclude the forum itself.

However, the most elegant function is to move the files and update the database with queries. It has a much smaller impact on a system than importing, namely the additional space it would eat up.

[Afrika]

OK thanks. What I have to change if the folder named picupload?

[y2ksw]

OK thanks. What I have to change if the folder named picupload?
Depending on what you have decided to do, you have to change the plugin code or move the files.

Code changes: The code is commented and uses self-explanatory variables, thus you should be able to find easily the place where to skip the forum path test and comment that part out or change it to your needs. Alternatively, you may hire somebody (not me) who is going to make the changes.

Move files manually: use shell commands and phpmyadmin for the queries ;)

[final kaoss]

I imagine this would work fine with vb3 & vb5?

[y2ksw]

I imagine this would work fine with vb3 & vb5?
Sorry, no. There was a 3.x version and there is no 5.x version and probably never will be.

[final kaoss]

Thanks, is there a way to make it only import http images (Non SSL images)? That is, skip any images with https in it and only download the "mixed content" http images.

[final kaoss]

I noticed that sometimes this plugin corrupts the filenames. Any fix for that?

[TTayfun]

please release 3.8.11 for vbulletin.

[ForumExcellence]

Thanks, is there a way to make it only import http images (Non SSL images)? That is, skip any images with https in it and only download the "mixed content" http images.

In the settings for the plugin (Settings -> Options -> vB4 Import External Images), put "https://" (without quotes) on a new line in the "Ignore" section, like so:

157170

You may consider downloading everything though, if you have the space/bandwidth to support it, because otherwise, external hosts could get rid of their images - either due to age, or some ulterior motive, and by surprise, like Photobucket.

[Piloux]

Hi and thanks for your efforts.

I just have a question, I noticed that the imported folder has to be chmod to 777, however all the imported files are automatically chmod to 644.

In this case, if a user posts an "infected" image that contains a php script or malicious code, are they blocked by chmod 644 or is there still a chance that the code gets executed when the hacker opens the url mysite.com/images/imported/infectedfile.png ?

Thanks

[Hostboard]

CHMOD/CHOWN are server based permissions and have no bearing to the end user if the file is served to them. What you need is to have an addon that uses FProt or ClamAV to scan attachments and imported files. I know there was Fractalizer for VB 3 and there was a VB4 variant but I think they were only for attachments. This is something that would need to be coded from scratch.

[Piloux]

CHMOD/CHOWN are server based permissions and have no bearing to the end user if the file is served to them. What you need is to have an addon that uses FProt or ClamAV to scan attachments and imported files. I know there was Fractalizer for VB 3 and there was a VB4 variant but I think they were only for attachments. This is something that would need to be coded from scratch.

OK thanks for your fast reply, that means that this plugin is pretty insecure? In this case is there a way to put an .htaccess in the imported folder that would deny anyone from running php scripts?

[Hostboard]

The plugin in itself is not insecure as I do not believe that the intention of scanning files for malicious code was ever its intent. That being said you can then say vBulletin is not secure in that the way it allows you to have attachments and upload avatars. There is a known exploit:

The following issues exist in vBulletin itself, reported by us to vBulletin support over 60 days ago(years now). Since vBulletin has not patched or disclosed the issues since that time, we now do so here:
An image decompression bomb vulnerability exists when vBulletin Options > Message Attachment Options > Resize Images = Yes. Disable it to protect your site.
An image decompression bomb vulnerability exists when allowing user uploads for avatars and profile pictures. To protect your site, change your forum's permissions so that users cannot upload custom avatars or profile pics.
An image decompression bomb vulnerability exists when using ImageMagick for images and allowing uploads. Currently known issues are for PDFs and TIFFs; however, because the filename of the incoming upload is not trustworthy, removing entries from the Attachment Manager or changing Attachment Permissions are not viable options. The following mitigation options exist:
Change vBulletin Options > Image Settings > Image Processing Library = GD, OR
Change your forum's permissions so that no users can upload anything.

In an attempt to figure the best way to resolve this sort of thing would be to update the Fractalizer Plugin to work with VB 4: https://vborg.vbsupport.ru/showthread.php?t=187482

[Hostboard]

Any fix for PHP 7.1+

PHP Fatal error: [] operator not supported for strings on line 324 in /home/........./public_html/forums/includes/cron/iei_cron.php

[shka]

Edit
if(!count($image_tags))
{
$image_tags[] = 'img';
}

to

if(!count($image_tags))
{
$image_tags = [] ;
$image_tags[] = 'img';
}

Or better to
if(!count($image_tags))
{
$image_tags = ['img'];
}

so the following code line numbers are the same. I expect there are more than one fatal errors in.


Edit:
former edit of line 342 was stupid. Deleted

[Hostboard]

Thank you for the quick reply :)
File edits made. Will check the log file latter on tonight and see if anything pops back up.

Still getting (1) error with this...

NOTICE: PHP message: PHP Fatal error: syntax error, unexpected ';', expecting ']' on line 334 in /home/........./public_html/forums/includes/cron/iei_cron.php"

This did not work...


if(!count($image_tags))
{
$image_tags = ['img'];
}


Trying the first notation...

[y2ksw]

I'm very unhappy with PHP 7, which converts the untyped language in a typed one. Having said this, you may test

if (isset($iei_ignore)) ...

in order to see if the variable was actually set, or better, use a different variable to read the value, and use

$iei_ignore = array();

beforehand. E.g.:

$iei_ignore = array();
$x_ignore = explode("\r\n", $input_ignore);
if (!is_array($x_ignore))
{
$iei_ignore[] = $bburl;
}
else
{
$iei_ignore = $x_ignore;
}

[shka]

Still getting (1) error with this...

NOTICE: PHP message: PHP Fatal error: syntax error, unexpected ';', expecting ']' on line 334 in /home/........./public_html/forums/includes/cron/iei_cron.php"

This did not work...


if(!count($image_tags))
{
$image_tags = ['img'];
}


Then you made a typing error. "syntax error, unexpected ';', expecting ']'" is not a php7 error or so. And you ar sure line 334? the qoted lines are in 324. Your quoted code can't be your inserted code.

But of course you can use the other version or from y2ksw (but the if conditions there is wrong I think)

[y2ksw]

$image_tags = array('img');

As far as I know, implicit arrays are not allowed for a long time by now, but explicit arrays like above, must work.

[Hostboard]

Is there anyway to get this to support a URL such as:


https://scontent-lax3-1.xx.fbcdn.net/v/t1.0-9/fr/cp0/e15/q65/94475070_10213603321997797_6910232200327725056_o.j pg?_nc_cat=108&_nc_sid=ca434c&efg=eyJpIjoidCJ9&_nc_oc=AQlsUa106dvp0CMIRD8nGvIE-2Hc9WyOAf3nmJ0ANiOGTbUAnpGlycnsQFC5Sbpeq-4zdhQUmJqUb9BVRXAP4Uvg&_nc_ht=scontent-lax3-1.xx&_nc_tp=14&oh=ee495fa093d4139e821daa867b450a8e&oe=5EC82DE0


If I try it comes back as invalid as I suspect it is looking for an image extension such as .jpg, .png, etc.

[djbaxter]

I was going to suggest you strip off everything after the .jpg extension so it would just be this:

https://scontent-lax3-1.xx.fbcdn.net/v/t1.0-9/fr/cp0/e15/q65/94475070_10213603321997797_6910232200327725056_o.j pg

but that URL gives an error: Bad URL timestamp

Maybe because it's a CDN, possibly a private CDN?

[Hostboard]

I was going to suggest you strip off everything after the .jpg extension so it would just be this:

https://scontent-lax3-1.xx.fbcdn.net/v/t1.0-9/fr/cp0/e15/q65/94475070_10213603321997797_6910232200327725056_o.j pg

but that URL gives an error: Bad URL timestamp

Maybe because it's a CDN, possibly a private CDN?

Not sure, but it should be able to pull down the image or at least I was hoping it would...

[djbaxter]

Not sure, but it should be able to pull down the image or at least I was hoping it would...

Not if the server is misconfigured or the link isn't accessible to the public.

[Hostboard]

I can enter the entire URL and the image is displayed from multiple locations. But as you pointed out it is more than likely with the code after the named image.

[djbaxter]

Ah yes. So it does.

I know that some online news services use similar formats to discourage or prevent image scraping. You may need to just accept that such images won't be imported. Alternatively, you can download the image from your browser and upload it from your local disk as an embedded attachment.

[Hostboard]

Ah yes. So it does.

I know that some online news services use similar formats to discourage or prevent image scraping. You may need to just accept that such images won't be imported. Alternatively, you can download the image from your browser and upload it from your local disk as an embedded attachment.

Because of what my site is/does, it is not something that I can do. I am just trying to supply the tools. I'll probably just have to have a developer look at it and see if it can be coded to accept URLs of this nature. Thanx for the replies :)

[y2ksw]

Is there anyway to get this to support a URL such as:


https://scontent-lax3-1.xx.fbcdn.net/v/t1.0-9/fr/cp0/e15/q65/94475070_10213603321997797_6910232200327725056_o.j pg?_nc_cat=108&_nc_sid=ca434c&efg=eyJpIjoidCJ9&_nc_oc=AQlsUa106dvp0CMIRD8nGvIE-2Hc9WyOAf3nmJ0ANiOGTbUAnpGlycnsQFC5Sbpeq-4zdhQUmJqUb9BVRXAP4Uvg&_nc_ht=scontent-lax3-1.xx&_nc_tp=14&oh=ee495fa093d4139e821daa867b450a8e&oe=5EC82DE0


If I try it comes back as invalid as I suspect it is looking for an image extension such as .jpg, .png, etc.

These URL's normally don't work, because they are usually a sub-set of a larger service.

It would eventually work, if the part after, including ? is removed. However it looks like a temporary link anyway.

[BadgerDog]

Hello Giovanni..

Hope you have been well. It's been a long time and I'm not sure that you are still active here.

I've been using your Beta 1.0.6 for several years with vBulletin 4.1.3pl1 and it’s been work great!

I’ve had to update to bulletin 4.25 now and I’m not sure what’s gone wrong, but the attachment files read by 1.0.6 into directory /images/imported are now showing with only a “?” where the picture should be?

When I check the link for any pic, it’s showing as an example.

/attachment.php?attachmentid=121951&d=1639421597

…. when it should be linking to a pic captured by your mod and stored within /images/imported?

I have the directory in your "Vb Import External Images" options page set to /images/imported.

I’m not sure what else I need to do to get into work with vBulletin 4.2.5 as it always has with 4.13pl1?

Can you help at all?

Thank you…

Regards,
Doug

[y2ksw]

Hello Giovanni..

Hope you have been well. It's been a long time and I'm not sure that you are still active here.

I've been using your Beta 1.0.6 for several years with vBulletin 4.1.3pl1 and it?s been work great!

I?ve had to update to bulletin 4.25 now and I?m not sure what?s gone wrong, but the attachment files read by 1.0.6 into directory /images/imported are now showing with only a ??? where the picture should be?

When I check the link for any pic, it?s showing as an example.

/attachment.php?attachmentid=121951&d=1639421597

?. when it should be linking to a pic captured by your mod and stored within /images/imported?

I have the directory in your "Vb Import External Images" options page set to /images/imported.

I?m not sure what else I need to do to get into work with vBulletin 4.2.5 as it always has with 4.13pl1?

Can you help at all?

Thank you?

Regards,
Doug
Hello Doug

In fact I have more or less moved away from web and forums, but I try to keep up - you never know.

I'm not sure what might have happened, but I guess, something at the server side has changed ... or the page encoding, style, PHP version, fopen permissions ... or the script was copied incomplete or was modified, or the database has a different encoding. Have you tried to change the setting and and change it again, in order to store it as a "new" setting?

[Old-Git]

Seeking a wee bit of advice of this mod. I'm in the process of rebuilding a vbforum from an old website that is now defunct. The old website ran this mod as an addon and it proved very useful. However, the restored database that I'm preping is currently running on a local machine on my home network so the URLs have all changed from FQDN to an IP address.

I've found a table in the database called IEI_IMG and it seems to me that all I need to do here is do a search and replace on the NewURL field and replace the sitename.com with IP address of the local machine on which it is running. Can anyone confirm that this is all I need to do to point the URL's for all the imported images to a new path on my local machine? Or will I have to single-step through every thread and replace every single URL?

[y2ksw]

Seeking a wee bit of advice of this mod. I'm in the process of rebuilding a vbforum from an old website that is now defunct. The old website ran this mod as an addon and it proved very useful. However, the restored database that I'm preping is currently running on a local machine on my home network so the URLs have all changed from FQDN to an IP address.

I've found a table in the database called IEI_IMG and it seems to me that all I need to do here is do a search and replace on the NewURL field and replace the sitename.com with IP address of the local machine on which it is running. Can anyone confirm that this is all I need to do to point the URL's for all the imported images to a new path on my local machine? Or will I have to single-step through every thread and replace every single URL?

I suggest to add your domain to the hosts file and access your local site as if it was remote.

[Old-Git]

Thanks for your reply Giovanni, I got it sorted in the end by doing a string replace in the database. The IEI_IMG table was just a simple record of what had been imported and from where, this was invaluable in helping me rebuild missing content so good call in including it in the package!

Is there any chance that this mod can be ported to VB5x, it's absolutely invaluable on my site!

[y2ksw]

Thanks for your reply Giovanni, I got it sorted in the end by doing a string replace in the database. The IEI_IMG table was just a simple record of what had been imported and from where, this was invaluable in helping me rebuild missing content so good call in including it in the package!

Is there any chance that this mod can be ported to VB5x, it's absolutely invaluable on my site!

The only way I could attempt to do this job for vB5, is to write a PHP or system/Bash or Python script for Linux, since vB5 has removed most of the features and hooks vB4 had. It requires you to have access to the system, of course.

Is there anybody else interested in such a solution?

[y2ksw]

I am at a Python script ... learning a new programming language :)

[Old-Git]

Love top see the result Giovanni, especially if it provides a VB5 implementation for this feature/addon!

[mp3president]

I would love to have this working on VB5.
I miss this!!!!

Yilmaz has made one based on your mod. But for some reason i can't get it to work!

It would be awsome!!!

[Dr.CustUmz]

@y2ksw I was messing around with this trying to get it to work on vb3, I managed to find an old copy off the vb3 version that does still work, but this one has more features. After reading EVERY page, it seems BOP has had no issues running this on vb3 as is, yet all my attempts have been unsuccessful.

Is this strictly for linux based environments, Or using WAMP (windows) would modifying the paths be sufficient enough to get this operating?

[y2ksw]

@y2ksw I was messing around with this trying to get it to work on vb3, I managed to find an old copy off the vb3 version that does still work, but this one has more features. After reading EVERY page, it seems BOP has had no issues running this on vb3 as is, yet all my attempts have been unsuccessful.

Is this strictly for linux based environments, Or using WAMP (windows) would modifying the paths be sufficient enough to get this operating?

It should actually work everywhere and PHP should translate the paths accordingly. If your web server is located on the C: drive, then a path starting with / should refer to C:\ automatically. In addition, paths in Windows are not case sensitive. The only issue I have had with windows web servers, were the read and write permissions, but adding the Everybody user with Modify rights fixed it.

[y2ksw]

Love top see the result Giovanni, especially if it provides a VB5 implementation for this feature/addon!

I am at it, but since I have moved my business from IT to agriculture, now my priorities are gardening :)

[Dr.CustUmz]

It should actually work everywhere and PHP should translate the paths accordingly. If your web server is located on the C: drive, then a path starting with / should refer to C:\ automatically. In addition, paths in Windows are not case sensitive. The only issue I have had with windows web servers, were the read and write permissions, but adding the Everybody user with Modify rights fixed it.

Thanks for the quick response, Ill play with it a little more on my environment and see if I can trace exactly whats going on. I'll post back with an update once I figure something out.