Import External Images

[TheLastSuperman]

Question though... what if its a 504 Gateway timeout, not from your host but for example if they are trying to link to an image that does not exist on imageshack or similar... does the clear.gif still replace properly then? Or does the gateway timeout hinder that from happening and produce these database errors?

Basically it's not his server ultimately timing out... it is but it isn't if that makes sense, due to the fact its trying to get an image from a server where the image does not exist then timing out.

[gsmlover4u]

Question though... what if its a 504 Gateway timeout, not from your host but for example if they are trying to link to an image that does not exist on imageshack or similar... does the clear.gif still replace properly then? Or does the gateway timeout hinder that from happening and produce these database errors?

Basically it's not his server ultimately timing out... it is but it isn't if that makes sense, due to the fact its trying to get an image from a server where the image does not exist then timing out.

sir you mean to stay that clear.gif not availeable on my server ?

[smirkley]

I have run this mod since, I think maybe a couple of weeks ago?

Everything has been running, I have set it pretty low to spread out the initial uploads over time.

I have noticed it hung somewhere, but it has stopped uploading, as in one image intensive thread,... it did the first image, but it hasnt done the followup posts in that thread. It should have gone through that date fairly quickly?

It has been days since the last upload to my image directory, and I know it isnt done.

Any idea of where to look at what hangs it up?

[gsmlover4u]

I have run this mod since, I think maybe a couple of weeks ago?

Everything has been running, I have set it pretty low to spread out the initial uploads over time.

I have noticed it hung somewhere, but it has stopped uploading, as in one image intensive thread,... it did the first image, but it hasnt done the followup posts in that thread. It should have gone through that date fairly quickly?

It has been days since the last upload to my image directory, and I know it isnt done.

Any idea of where to look at what hangs it up?

run this hack manualy from scheduled task manager and see what happend

[smirkley]

run this hack manualy from scheduled task manager and see what happend

When I manually run the cron, all I get is a blank page with the title "vB4 Import External Images" at the top. Nothing else.

[Nirjonadda]

I have changed domain name of my Web site and now old domain imported image link are not working,How i can get link working and old domain link replace to new domain link?

[y2ksw]

Question though... what if its a 504 Gateway timeout, not from your host but for example if they are trying to link to an image that does not exist on imageshack or similar... does the clear.gif still replace properly then? Or does the gateway timeout hinder that from happening and produce these database errors?

Basically it's not his server ultimately timing out... it is but it isn't if that makes sense, due to the fact its trying to get an image from a server where the image does not exist then timing out.
Yes, if the local server has not died by then. And if it has died, it would repeat querying. With CURL, things are slightly better, since it is possible to tune the timeouts in the options.

If database errors occur, the the db server has died before the web service, and this should never happen. In fact, in production sites the database timeout is one day, because then the connection will drop when the web server drops the page, and at no moment earlier.

[y2ksw]

When I manually run the cron, all I get is a blank page with the title "vB4 Import External Images" at the top. Nothing else.
We have had this particular issue with another user of this plugin. In the end we worked out that extremely large images would drop PHP without error and warning, and it could be fixed only by adding more RAM to the single processes.

[y2ksw]

I have changed domain name of my Web site and now old domain imported image link are not working,How i can get link working and old domain link replace to new domain link?
UPDATE post SET pagetext=REPLACE(pagetext, 'old url', 'new url')

and similarily everywhere else you need to (signatures, blogs, cms)

Also, you should empty the postparsed table.

As always, please make a backup before issuing this query.

[smirkley]

We have had this particular issue with another user of this plugin. In the end we worked out that extremely large images would drop PHP without error and warning, and it could be fixed only by adding more RAM to the single processes.

Question, how do I do that?

And also, how can I find out which image it is that is causing this?

[smirkley]

I am attempting to find out where it stalled out at using phpmyadmin, when I noticed something strange...

The replacement url for missing images looks like this,.. (sometimes, not all instances)

h ttp://www.mysite.com//images/misc/BrokenImage.gif
(I added a space in http to prevent this url from parsing here)

edit... figured it out, my bad, I had added a leading / to the broken image image url in acp

now just to figure out how to correctly convert it without having to edit each and every field lol

[tommythejoat]

My moderators are reporting problems with this feature. It appears that images are being replaced with the wrong, seemingly random image.

I spent some time looking at the iei_img table and the post table with the following results.

We have 798,156 posts.
23,105 of the posts contain at least one IMG bbcode.
25,859 are marked iei_parsed.

In the iei_img table we have 34,257 records.
Of these 20,420 saved images are mapping unique external images.
We have 11,015 records with our web root (forum) directory as the saved entry (no jpg)
The big problem is 911 of the save images map multiple external images with as many as 204 references down to 2 references.

Other than restoring my backups from before I installed iei, I do not know how to recover the erroneous image mappings.

I do not really understand how the batches are managed while importing images. It appears that it is not working correctly.

We are running 4.2.1 with iei 1.0.5

[smirkley]

I too have reservations on a reverse-implentation plan. Is there a method of reversing ones way out of this mod in the event it becomes undesirable in the end?

But most importantly, I wish to make this mod fully implementable as the concept is important to me.

[y2ksw]

My moderators are reporting problems with this feature. It appears that images are being replaced with the wrong, seemingly random image.

I spent some time looking at the iei_img table and the post table with the following results.

We have 798,156 posts.
23,105 of the posts contain at least one IMG bbcode.
25,859 are marked iei_parsed.

In the iei_img table we have 34,257 records.
Of these 20,420 saved images are mapping unique external images.
We have 11,015 records with our web root (forum) directory as the saved entry (no jpg)
The big problem is 911 of the save images map multiple external images with as many as 204 references down to 2 references.

Other than restoring my backups from before I installed iei, I do not know how to recover the erroneous image mappings.

I do not really understand how the batches are managed while importing images. It appears that it is not working correctly.

We are running 4.2.1 with iei 1.0.5
If the "missing/invalid" image replacement is disabled, the original links are maintained, when this product is unable to import an image. For this reason it is most probable that random images are still within the img tags, since those are unlikely to work, ever. For example, some sites don't allow hot linking, and thus put junk redirects on line. Then this product wants to import them and sees, it can't. If missing images are replaced, it will typically use clear.gif or something alike to replace the missing image. Else, it will leave the link intact, and it may work for a while, until this kind of images disappear from source.

There is nothing I can do about it, this is how Internet works.

You may reparse all or part of the posts setting the iei_parsed flag to 0. Some sites do allow hot linking after a while and so you may get them later. I've been using this product over many years now on large production sites and we have had a number of problems, which were all related to remote images. Some of which were WORM infected, some where hotlink protected and others just would not conform to the standards. But these are few exceptions out of thousands of imported images every month.

[y2ksw]

I too have reservations on a reverse-implentation plan. Is there a method of reversing ones way out of this mod in the event it becomes undesirable in the end?

But most importantly, I wish to make this mod fully implementable as the concept is important to me.
Eventually you could parse the old/new urls from the iei_parsed table and attempt to replace with global queries the links. On large forums this could be however a very time and resource consuming operation. And in any case, it is the only one you could use, since posts may be copied to any place and it is impossible to tell if the contained links came from your site or anywhere else.

[tommythejoat]

I think perhaps you missed my point on the multiples. I just searched my forum posts for "/2007/01/1.jpg" which should yield at most a few copies of the same image in related posts. Instead it returns 558 hits in a total of 371 posts.

I don't see any way for this to occur without an indexing error in the software. I think it is at least worth a look at the arbitrary name generator and where it stores its counter.

[y2ksw]

I think perhaps you missed my point on the multiples. I just searched my forum posts for "/2007/01/1.jpg" which should yield at most a few copies of the same image in related posts. Instead it returns 558 hits in a total of 371 posts.

I don't see any way for this to occur without an indexing error in the software. I think it is at least worth a look at the arbitrary name generator and where it stores its counter.
Double or more hits in a single post is possible, if for instance somebody has pasted an image N times. This happens often in my forums, where people copy their preferred smilie instead of asking me to add them to the stock items. Similarily, you'll find underlines, page separators, signatures and all kinds of graphical additions people just paste on the fly without even knowing it.

In the case of having 2 or more hits of an image in a single post, it would be interesting to see if there are actually two or more copies of that image. If it isn't, then maybe something strange is going on.

[tommythejoat]

You did notice that I said 558 copies of the same image in a total of 371 posts.

Yes it would be perfectly reasonable to see several instances of an image. It can happen when a post is quoted for example or a person might even refer to the same image in two or three different posts. When it gets to the hundreds it cannot be due to random selection by users, it can only be a bug.

If one has left the replacement image value blank as I have done, what should we expect to see? We have lots of entries in the replacement image of http://mysite/ with no file reference in the URL. I have been reading those as meaning the image was missing from the source site.

[smirkley]

I have fixed my first problem extra "/" in replacement url,... caused when I added incorrectly the leading slash in my replacement image.

Need help with my second problem.

Operation has hung and I cannot figure out how to identify the image that it is hung on.

[y2ksw]

I have fixed my first problem extra "/" in replacement url,... caused when I added incorrectly the leading slash in my replacement image.

Need help with my second problem.

Operation has hung and I cannot figure out how to identify the image that it is hung on.
It is the first post with an img tag which has not the iei_parsed flag set to 1.

SELECT * FROM post WHERE iei_parsed=0 AND pagetext LIKE '%[img%' ORDER BY postid LIMIT 1

[y2ksw]

You did notice that I said 558 copies of the same image in a total of 371 posts.

Yes it would be perfectly reasonable to see several instances of an image. It can happen when a post is quoted for example or a person might even refer to the same image in two or three different posts. When it gets to the hundreds it cannot be due to random selection by users, it can only be a bug.

If one has left the replacement image value blank as I have done, what should we expect to see? We have lots of entries in the replacement image of http://mysite/ with no file reference in the URL. I have been reading those as meaning the image was missing from the source site.
If a post is copied, it would count as a different post.

Before I step to say it's a bug, let us see what actually happens. How do you count the occurrences of a single image?

If the replacement image is empty and you have a replacement enabled, probably you will end up with a home URL only. I haven't tried that yet :)

[tommythejoat]

If a post is copied, it would count as a different post.

Before I step to say it's a bug, let us see what actually happens. How do you count the occurrences of a single image?

If the replacement image is empty and you have a replacement enabled, probably you will end up with a home URL only. I haven't tried that yet :)

I am examining the iei_img table and count the occurrences of oldurl grouped by newurl. I included the max and min oldurl to demonstrate the fact that they are different. In fact if all are displayed, they are generally all different. It is as though the background batch process were not advancing the newurl calculation. Perhaps it would work if I were keeping the oldurl file names.

Here is the result for the first few rows of the query. Note the url references are all public and do not need to be obscured.

[smirkley]

It is the first post with an img tag which has not the iei_parsed flag set to 1.

SELECT * FROM post WHERE iei_parsed=0 AND pagetext LIKE '%[img%' ORDER BY postid LIMIT 1

Thank you, that gave me the clue I needed.

I found the post, but what I found may be indicating a bug.

My instance of where it hung:

Put an [img ] "http://yourpic.com" [/img ] in your sig. good luck!

The above quote is the actual text in the post where it hung, and in this case it was a poster some years back giving instruction on how to use the IMG tag in the signature.

Please note the poster put an added space in the IMG open and close tag to keep it from parsing, in their example of instruction.

It appears that this mod still looks at the text "[img ]" as "[img]" instead and attempts to do a replacement on this even though it is not an actual IMG tag to be replaced.


edit to add: I cleared this problem, and it immediatly hung on the next instance of where "[img] was used in a post as text, but without the closing tag. Again, another instance of a poster giving instruction.

[tommythejoat]

To better understand what might be going on, I have been reading the iei_cron.php file.

My php reading ability is not really up to production level, but I can sort of make things out. I noticed that you were using php image functions and we have ImageMagick installed on our site and selected in the vBulletin options.

I was wondering if that might create a context that had not really been tested with this mod.

Another source of stress may be that we have been adding content to the site since 2001 and have leftover fragments from ubb, infopop and bbv2 implementations. We moved to vBulletin with 3.5 in 2008 and imported all the content we could. There are lots of old broken references to previous servers in our posts.

[smirkley]

y2ksw I was just wondering if you had a chance to checkout my last post above to see if it is a bug or something.
I know you may be busy and I can wait. Just was wondering.

tnx

[y2ksw]

To better understand what might be going on, I have been reading the iei_cron.php file.

My php reading ability is not really up to production level, but I can sort of make things out. I noticed that you were using php image functions and we have ImageMagick installed on our site and selected in the vBulletin options.

I was wondering if that might create a context that had not really been tested with this mod.

Another source of stress may be that we have been adding content to the site since 2001 and have leftover fragments from ubb, infopop and bbv2 implementations. We moved to vBulletin with 3.5 in 2008 and imported all the content we could. There are lots of old broken references to previous servers in our posts.
This is almost at 100% one of the problems. The image importer relies on GD only. Do you have also GD installed?

[tommythejoat]

I don't think there is a security issue in showing my config.

'./configure' '--disable-fileinfo' '--disable-pdo' '--enable-bcmath' '--enable-calendar' '--enable-exif' '--enable-ftp' '--enable-gd-native-ttf' '--enable-libxml' '--enable-magic-quotes' '--enable-mbstring' '--enable-soap' '--enable-sockets' '--prefix=/usr/local' '--with-apxs2=/usr/local/apache/bin/apxs' '--with-bz2' '--with-config-file-path=/usr/local/lib' '--with-config-file-scan-dir=/usr/local/lib/php.ini.d' '--with-curl=/opt/curlssl/' '--with-curlwrappers' '--with-freetype-dir=/usr' '--with-gd' '--with-imap=/opt/php_with_imap_client/' '--with-imap-ssl=/usr' '--with-jpeg-dir=/usr' '--with-kerberos' '--with-libdir=lib64' '--with-libexpat-dir=/usr' '--with-libxml-dir=/opt/xml2' '--with-libxml-dir=/opt/xml2/' '--with-mcrypt=/opt/libmcrypt/' '--with-mysql=/usr' '--with-mysql-sock=/var/lib/mysql/mysql.sock' '--with-mysqli=/usr/bin/mysql_config' '--with-openssl=/usr' '--with-openssl-dir=/usr' '--with-pcre-regex=/opt/pcre' '--with-pic' '--with-png-dir=/usr' '--with-tidy=/opt/tidy/' '--with-xmlrpc' '--with-xpm-dir=/usr' '--with-xsl=/opt/xslt/' '--with-zlib' '--with-zlib-dir=/usr'

Here are the GD details

gd
GD Support enabled
GD Version bundled (2.0.34 compatible)
FreeType Support enabled
FreeType Linkage with freetype
FreeType Version 2.3.11
GIF Read Support enabled
GIF Create Support enabled
JPEG Support enabled
libJPEG Version 6b
PNG Support enabled
libPNG Version 1.2.49
WBMP Support enabled
XPM Support enabled
XBM Support enabled

Directive Local Value Master Value
gd.jpeg_ignore_warning 0 0

[y2ksw]

I don'r see a problem with it, everything is in place.

[smirkley]

Yes just verified the 10 instances on my board that the mod cant recognise the difference between valid and invalid image tags in posts.

invalid meaning - [img] or [ img] used in conversation inside the post that is either not with a closing tag, or is standalone or has a space added inside the tag to keep it from parsing.

I am editing the instances where they occur on my board just to make it work.

[y2ksw]

Yes just verified the 10 instances on my board that the mod cant recognise the difference between valid and invalid image tags in posts.

invalid meaning - [img] or [ img] used in conversation inside the post that is either not with a closing tag, or is standalone or has a space added inside the tag to keep it from parsing.

I am editing the instances where they occur on my board just to make it work.

Right. These tags will not be recognized ever. They would not load an image either. There is no way to tell if they belong to an image or not.

[korny]

Not sure if this was asked before, but will this work for 3.8.7? If not, do you have one developed similar to this for 3.8.7?

[y2ksw]

Not sure if this was asked before, but will this work for 3.8.7? If not, do you have one developed similar to this for 3.8.7?
https://vborg.vbsupport.ru/showthread.php?t=123338

It was actually in the description :)

[korny]

Ha ha, oops. Thanks Giovanni!

That hack isn't really the same as this one is it? This one here takes the image and stores it in the file system. The other hack you linked just caches it right?

[y2ksw]

Ha ha, oops. Thanks Giovanni!

That hack isn't really the same as this one is it? This one here takes the image and stores it in the file system. The other hack you linked just caches it right?
It is not exactly the same. But it still saves the images locally to your server, even though it does not create subfolders.

[Morrus]

Wow; clever idea! I wonder if it's possible to do that with files that aren't images? PDFs, .docs, zips, etc? One thing my forum has is a lot of old dead links to files that aren't there any more.

[y2ksw]

Wow; clever idea! I wonder if it's possible to do that with files that aren't images? PDFs, .docs, zips, etc? One thing my forum has is a lot of old dead links to files that aren't there any more.
Yes, pretty easy. It would however require another product based off from this one.

[gsmlover4u]

can we change the name of images directory my images import in import directory now can i change directory name import to anyone

[y2ksw]

can we change the name of images directory my images import in import directory now can i change directory name import to anyone
You can change the name of the image directory any time. However, you can't change the name of the image directory on the server, because all the posts before the change will continue to point to the "old" folder. So if you change the directory name in the options from oldfolder to newfolder, you will have to keep both oldfolder and newfolder on the server.

[fariborz khan]

if I change my domain name what I must do?
I like this mod
mark as installed

[y2ksw]

if I change my domain name what I must do?
I like this mod
mark as installed
Essentially you would have to run a query to replace the references in your posts like:
UPDATE post SET pagetext=REPLACE(pagetext, '1st.com', '2nd.com') or fix the issue in your .htaccess file if you still have both domains pointing to the same webspace.

[webmastersitesi]

How this system work? when will be images imported? I imported product nothing happened current images

[y2ksw]

It'll take some time to see what happens, because it will handle a number of posts each time the cronjob runs.

[tommythejoat]

We upgraded our hardware and system software and moved to 4.2.2. Since then this has stopped working.

The symptom is that the inserted image gets replaced as iei runs, but instead of the reference to the imported image we get a reference to the web root.

It has taken some time for the users to notice and complain and then it took me a while to see what they were complaining about.

The iei_img table has about 9,000 references that are mapped to web root. Perhaps that is the placeholder while they are being processed.

We also have another 45 images that map to 2014/01/1.jpg and there is not 2014 folder in the imported images. I would appreciate any insight into why the 2014 images are not coming in. Did something happen as of the first of the year?

[tommythejoat]

Another possible problem is that many sites are now watermarking their images with PHP and GD dynamically and posting a URL like http://www.fhwatch.com/watermark.php?image=iPW10779a07.jpg. Will iei handle that kind of reference?

The output stream is a valid image as long as the mime type is recognized.

http://www.fhwatch.com/watermark.php?image=iPW10779a07.jpg

[y2ksw]

The links should work. Does iei create new folders on your server?

[tommythejoat]

There are folders going back to 2001 and lots of images. It stopped creating folders for January 2014. There are 65 diffferent references in iei_img to 2014/01/1.jpg so they cannot all be references to the same image. How can I find out where they were harvested from? Can I rely on them being in date order among the posts? I can search for the 2014/01/1.jpg reference and maybe they are in the same orderr as the iei_img table entries.

It is a new server, I am guessing that the cron job runs as vb itself and there might be permission issues on the parent folder /image/imports.

[y2ksw]

If it stopped creating folders in 2014 something has changed. I would suspect that the access rights of the web service can't create folders any more. Since holidays are always the moment where incisive changes take place, maybe your hosting provider has changed something in this area. Or maybe an FTP file permission slipped unintentionally through your hands :)

Normally, the folder containing the years subfolders should have chmod 755 or 777, depending on the server settings. If it has become for some reason 644 or similar, the directories are seen, but can't change or extend.

[tommythejoat]

All of the folders have 755 setting. We own the system at a colocation center. They do not maintain the software. We are running WHM/CPanel on Centos.

I think I am missing something about the philosophy of missing images. If the cron job cannot import an image, what does it put in the iei_img table?

It appears to check for an existing file and then increment the name.

The other thing I noticed is that we are getting image urls of the form http://mysite.net/watermark.php?file_identifier.php. Can your code fetch a file of that kind?

[y2ksw]

What it puts in place of missing images depends on the settings in the options. It may replace the image with a local image (e.g. clear.gif), or keep the link as-is (ignore).

It does check for the name and increments if already present in order to create a new set of images each month.

All links are imported. How they translate to your system however depend on the settings, and probably the *.php images have to be converted. If you decide to keep *.php "images", you will have to add a .htaccess rule for excluding php code execution for the imported files.

On my systems I have the file conversion always on (converto to JPG), because it also removes eventually present WORM's or other "crafted" images. Conversion will loose animations and transparency, but this is of much less concern. Crafted images may contain arbitrary runnable code on the client side.

[tommythejoat]

I will presume that we/I can figure out how to get images to actually upload to our server and actually fix the current problem. It may be related to problems with the image library since we have both GD and ImageMagick installed. It may have something to do with Curl since we also have PHP Soap installed.

If I delete the current repetitive local urls from iei_img and put the original url back in the posts in the order of the posts, and then set iei_parsed back to 0, will the cron job reprocess them?

I would ensure that the cron job can actually process the images and create the local files on my system first.

[y2ksw]

By setting iei_parsed to 0 the cronjob will reprocess the records.

[majjed2008]

Hello

Thank you for this products i like it .
regarding resizing it is default 0 , what is it Cm or PX or what exactly so i can use it in correct way

thank you

[y2ksw]

Hello

Thank you for this products i like it .
regarding resizing it is default 0 , what is it Cm or PX or what exactly so i can use it in correct way

thank you
px.

[Bob_R]

What I hate about this mod is that you cannot hotlink images.

My server people have tried (thru cPanel) and it appears it's because of this mod.

Does anyone have any comments on this?

Thanks!

EDIT: FYI - I have vBSEO installed

[y2ksw]

What I hate about this mod is that you cannot hotlink images.

My server people have tried (thru cPanel) and it appears it's because of this mod.

Does anyone have any comments on this?

Thanks!

EDIT: FYI - I have vBSEO installed
In fact the idea is to move entirely hotlinked images to your server. If this isn't what you want, then it is not a mod you should use :)

[Bob_R]

In fact the idea is to move entirely hotlinked images to your server. If this isn't what you want, then it is not a mod you should use :)

What are you talking about?

Is that supposed to represent an answer to my question?

I realize what this mod does. I use it and have for quite some time now.

So, it's YES you cannot hotlink images if you use this mod.

[RichieBoy67]

What I hate about this mod is that you cannot hotlink images.

My server people have tried (thru cPanel) and it appears it's because of this mod.

Does anyone have any comments on this?

Thanks!

EDIT: FYI - I have vBSEO installed

I'm a little confused as to what you mean here. Why are you going through cpanel to add links to your images?

Have you tried editing an individual post and creating the image link there?

[Bob_R]

I'm a little confused as to what you mean here. Why are you going through cpanel to add links to your images?

Have you tried editing an individual post and creating the image link there?

No sir not image link, hotlink.

[RichieBoy67]

I understand what a hotlink is but why do you want to import the images that are served on another server and then hot link them? it sort of defeats the whole purpose of this mod unless you mean something different then what I understand hotlinking to be.

[Bob_R]

I understand what a hotlink is but why do you want to import the images that are served on another server and then hot link them? it sort of defeats the whole purpose of this mod unless you mean something different then what I understand hotlinking to be.

I thought hotlinking said it all. I apologize. Maybe it's my bad. I'll explain what I'm talking about.

Someone links to a pic on my site to display it on their site. Instead of the pic they linked to they get this (see screenshot)

[RichieBoy67]

Oh I understand now. You are really trying to stop hotlinking. I thought you were trying to encourage it or allow it which seemed very odd to me. lmao

You could always disable right clicking if you have a major problem with it. I cannot even tell you how much bandwidth gets wasted on my site from hotlinking but most of the images on my site were posted on other sites anyways so we allow it.

I would like to use watermarking though and I am not sure if that would work with this mod either.

Sorry I cannot answer your question. Glad I know what you mean now though. Great concept there!

[y2ksw]

What are you talking about?

Is that supposed to represent an answer to my question?

I realize what this mod does. I use it and have for quite some time now.

So, it's YES you cannot hotlink images if you use this mod.
As it turns out, the question was different. You want to know if you can block hotlinking on your server. The answer is yes and it has nothing to do with this mod.

In order to block hotlinking you will have to add 3 (or more) lines in your .htaccess file, like:

RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?yourdomain.com(/)?.*$ [NC]
RewriteRule .*\.(gif|jpg|jpeg|bmp)$ - [F,NC]

[Bob_R]

As it turns out, the question was different. You want to know if you can block hotlinking on your server. The answer is yes and it has nothing to do with this mod.

In order to block hotlinking you will have to add 3 (or more) lines in your .htaccess file, like:

RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?yourdomain.com(/)?.*$ [NC]
RewriteRule .*\.(gif|jpg|jpeg|bmp)$ - [F,NC]

OK, maybe I'll try it by adding the code to the .htaccess myself. My server people tried it thru cPanel and it did not work. Not saying that it's this mods fault just sharing.

I also heard from a trusted friend that "hotlink protection (via an htaccess file) doesn't protect images uploaded to vbulletin because they are being requested via php files, not via their native file name."

I'll keep you informed.

[sticky]

This product struggles with importing images that have anything after the .jpg

Would you be able to make it so files with strings like this: cmonmanjpg3Fw3D50626h3D189-1.jpg%3fw%3d506%26h%3d189

Still import?

[massiveblack]

I installed as directed, then adjusted the settings and pushed save. I checked the ftp and no folder was made. The pdf instruction file says to put the php file in the root of the forums but the zip has it in the includes. I first put it in includes then imported the xml. nothing happens....no imported folder is created...nothing happens. i also tried it inside the root, per the pdf instructions. nothing happens. please advise. thank you!!

[y2ksw]

This product struggles with importing images that have anything after the .jpg

Would you be able to make it so files with strings like this: cmonmanjpg3Fw3D50626h3D189-1.jpg%3fw%3d506%26h%3d189

Still import?
Please set in the options "Always Convert Images" and/or uncheck "Use Original File Names" and all your troubles will go at once.

[y2ksw]

I installed as directed, then adjusted the settings and pushed save. I checked the ftp and no folder was made. The pdf instruction file says to put the php file in the root of the forums but the zip has it in the includes. I first put it in includes then imported the xml. nothing happens....no imported folder is created...nothing happens. i also tried it inside the root, per the pdf instructions. nothing happens. please advise. thank you!!
Please make sure you understand how to install vbulletin products. There are several guides on this site and installing mine is not more difficult than installing any other product.

The imported folder is not created and as stated in the guide, you will have to choose or make one on your own and then verify it has the access rights to write new files and folders.

And, finally, it may take quite some time before it creates images, since it will parse all posts and unless it does not find any, it will not create anything.

[massiveblack]

Ok thank you. Yes I do understand, but the language of the instructions was not clear and suggested that it was an automated process somehow. The folder is made and the permissions are set. I will check in and see if it is working. Thanks for the prompt reply.

[RichieBoy67]

The imports are an automated process but you need to create a directory for the imported images and enter the path so the process knows where to place the imported images. :)

[Manoel J?nior]

You can import images directly to my database rather than to a folder on my FTP?

[majjed2008]

hello

can we made it import the VB image also from thumps like that we have all picture in just onne folder ?

Thank you

[y2ksw]

You can import images directly to my database rather than to a folder on my FTP?
No. It is not intended to create attachments.

[y2ksw]

hello

can we made it import the VB image also from thumps like that we have all picture in just onne folder ?

Thank you
It is not possible to determine if an image you see in the posts is a thumb or actually an image. Thus it will import only images which have a recognizable format with the IMG BBCode.

[majjed2008]

It is not possible to determine if an image you see in the posts is a thumb or actually an image. Thus it will import only images which have a recognizable format with the IMG BBCode.

thank you is there any way that we can import thump or change them to be in folder as image , anyone know please ?

[elfenny]

I created the folder with 777 permissions but not create subfolders.

The mod does not work, how can I fix?

[Bob_R]

As it turns out, the question was different. You want to know if you can block hotlinking on your server. The answer is yes and it has nothing to do with this mod.

In order to block hotlinking you will have to add 3 (or more) lines in your .htaccess file, like:

RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?yourdomain.com(/)?.*$ [NC]
RewriteRule .*\.(gif|jpg|jpeg|bmp)$ - [F,NC]

OK, maybe I'll try it by adding the code to the .htaccess myself. My server people tried it thru cPanel and it did not work. Not saying that it's this mods fault just sharing.

I also heard from a trusted friend that "hotlink protection (via an htaccess file) doesn't protect images uploaded to vbulletin because they are being requested via php files, not via their native file name."

I'll keep you informed.

OK, did the code. It works if you post an image to another site with HTML img.

But, all I have to do is wrap the image in BB code ex. www.thisisanimage and it still shows on the offending site.

Anyway around this?

Thanks!

[Bob_R]

OK, I believe it is working. Because of this mod though you cannot use a hotlnk error image because it copies the hotlink error image and replaces the original pic on your site.

[y2ksw]

OK, I believe it is working. Because of this mod though you cannot use a hotlnk error image because it copies the hotlink error image and replaces the original pic on your site.
There is no way around. Sites may enable hotlink protection at their discretion, but usually the plugin works. Some exceptions made. You may "whitelist" those domains in the options and thus they will keep the hotlink.

[Bob_R]

There is no way around. Sites may enable hotlink protection at their discretion, but usually the plugin works. Some exceptions made. You may "whitelist" those domains in the options and thus they will keep the hotlink.

It's working except for that for the the hotlink image also ends up showing on my site.

I've since taken out the image. It'll now just show a blank empty post on the offending site.

I'm I reading you right, whitelist will make the image show? How do I whitelist?

Thanks.

[y2ksw]

It's working except for that for the the hotlink image also ends up showing on my site.

I've since taken out the image. It'll now just show a blank empty post on the offending site.

I'm I reading you right, whitelist will make the image show? How do I whitelist?

Thanks.
Please go to the options. There is a big field with according description and most probably already a "path" of a previous release of this product. Just add as many items you need ;)

[Bob_R]

Please go to the options. There is a big field with according description and most probably already a "path" of a previous release of this product. Just add as many items you need ;)

Which options, sir? I went into my vbulletin options for this mod and did not see as you say "a big field with according description and most probably already a "path" of a previous release of this product. Just add as many items you need."

[y2ksw]

It is the "Ignore" field.

[Bob_R]

It is the "Ignore" field.

So, just put the URL of a site or sites in the Ignore field and my hotlink error message WILL show on the offending site and NOT show on my site (on my site the hotlink image was replacing the image which was hotlinked to begin with)?

[y2ksw]

So, just put the URL of a site or sites in the Ignore field and my hotlink error message WILL show on the offending site and NOT show on my site (on my site the hotlink image was replacing the image which was hotlinked to begin with)?
If you insert the host in the ignore field, the images will not be imported. What they will show, in the end, depends on the site from where you try to show images. Which essentially may be hotlink blockers or anything else :)

[Bob_R]

I hope I'm using the proper terminology. Because you answered my question and now you're saying that this isn't the solution.

Here's the problem with this mod and hot linking (unless I'm doing something wrong but I tested it).

On MY site if I place a hot link image error to show up on the offending site which has grabbed an image from my site. The error image shows BUT the mod also replaces the image on my site with the error image.

Only want the error image to show on the offending site (obviously).

If I don't include an error image the offending site gets a blank screen and the image on my site stays intact. That's fine right? Shows it's working!

Thanks for your help!

[y2ksw]

I hope I'm using the proper terminology. Because you answered my question and now you're saying that this isn't the solution.

Here's the problem with this mod and hot linking (unless I'm doing something wrong but I tested it).

On MY site if I place a hot link image error to show up on the offending site which has grabbed an image from my site. The error image shows BUT the mod also replaces the image on my site with the error image.

Only want the error image to show on the offending site (obviously).

If I don't include an error image the offending site gets a blank screen and the image on my site stays intact. That's fine right? Shows it's working!

Thanks for your help!
This is simply impossible. If you show a hotlink image to other sites, your site must remain unaffected. This is a server issue and must be setup correctly in the .htaccess file, and never was an issue with the mod.

If the offending site gets a blank screen, then definitively your hotlink formula isn't working. Get Firebug for Firefox and check the response on the "missing" image, I'm almost 100% sure it's an error 5xx, which translates in a scripting error in .htaccess.

The mod does only download images from other sites, and if they have hotlinking images, it will download those, unless you have them excluded = ignored :)

[Bob_R]

This is simply impossible. If you show a hotlink image to other sites, your site must remain unaffected. This is a server issue and must be setup correctly in the .htaccess file, and never was an issue with the mod.

If the offending site gets a blank screen, then definitively your hotlink formula isn't working. Get Firebug for Firefox and check the response on the "missing" image, I'm almost 100% sure it's an error 5xx, which translates in a scripting error in .htaccess.

The mod does only download images from other sites, and if they have hotlinking images, it will download those, unless you have them excluded = ignored :)

Here's my .htaccess in regards to this situation

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?domain.com [NC]
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?domain.com/forums [NC]
RewriteRule \.(jpg|jpeg|png|gif)$ - [NC,F,L]

[y2ksw]

Here's my .htaccess in regards to this situation

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?domain.com [NC]
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?domain.com/forums [NC]
RewriteRule \.(jpg|jpeg|png|gif)$ - [NC,F,L]

Provided that domain.com is your domain, your hotlink protection must look this way. Please see the additional backslash, no second line needed and no fancy queries. It's your site and you know how it can be addressed, so make it short and simple. The missing additional backslash and unsupported fancy queries may cause a blank screen. The F directive does generate a failure, the rest should be shown correctly on the offending page:
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://www\.domain\.com [NC]
RewriteRule \.(jpg|jpeg|png|gif)$ - [NC,F,L]If you used this tool: http://www.htaccesstools.com/hotlink-protection/ for generation, it does not work correctly and it is a shame, but fortunately not mine.

The above rules says: If the referer is not blank and if it starts with http://www.domain.com then fail on these file extensions: .jpg, .jpeg, .png and .gif. This means in cleartext, if the referrer is blank or the referer starts with http://www.domain.com, then the images are shown. Direct page access will thus show the images, but if you are navigating the offending site, they will not.

I hope this extra lecture was exhaustive enough and has really nothing to do with the mod. :)

[shots]

Will this mod pickup changes?

i.e. if someone links an image and then a few days later changes the image (but not the filename) will this mod recognize that?

Followup: if so, will it overwrite the locally stored image or create an additional one with a -1 or something.

[y2ksw]

Will this mod pickup changes?

i.e. if someone links an image and then a few days later changes the image (but not the filename) will this mod recognize that?

Followup: if so, will it overwrite the locally stored image or create an additional one with a -1 or something.
The best way to know is to try it out. There are too many scenarios about this argument to efficiently give an answer. Generally, it imports images from external servers to local. What you do in local with these files, is your responsibility, but users have no influence. What people do on remote sites is beyond our possibilities :)

[ar15dcm]

Will it do the same in CMS articles? If not, what would it take to add that functionality?

[tbworld]

Will it do the same in CMS articles? If not, what would it take to add that functionality?

I read in the thread that the CMS is not supported. The post was dated 2012, so don't take my word on it. :)

[y2ksw]

Will it do the same in CMS articles? If not, what would it take to add that functionality?
It's not supported. It would take quite some efford to read the articles the same way as it does for the forum posts and apply the changes. Essentially it needs another cronjob with almost no shared code.

[vvlo]

It is very useful product and I've used it for years
I will soon be ready for vb 5
Do you plan to do this?

[y2ksw]

I haven't tried it yet, but it may work probably just out of the box or with very few changes in vbulletin 5.

[vvlo]

I haven't tried it yet, but it may work probably just out of the box or with very few changes in vbulletin 5.


I am very much looking forward to hearing good news
This is one of the most popular products that are offered by you

Sorry for my bad English

[Space_psy.be]

One thing I can confirm is that it works in v3.8.7-P4 without any changes.

For what I need it, I've done a small modification, mainly do a backup of the post before updating it.

What I'm most struggling with is the 27000+ posts that have images in them. It is slow. So I'm now rewriting parts of this addon so that it can run from cli.

If there is interest, I can contribute or publish the final cli version once it is finished. The code will not be nice though, I haven't coded much php in the last couple of years.

[madness85]

working great :) but if the thread is deleted does it clean up the un-used images?

[madness85]

No it doesn't clean un-used images :( can't use this sadly if it can't do it.

[y2ksw]

On large forums you can't tell if an image is reused. This is a plugin for typically very large forums, where the lack of images is much more critical than a few too many. Also, it may come handy to have a copy. Please think about it as a feature :)

[sticky]

By setting iei_parsed to 0 the cronjob will reprocess the records.


Where do I do this? I can't find iei_parsed

The product stopped working automatically for me today but when I run the cron job manually it works just fine.

[sticky]

Where do I do this? I can't find iei_parsed

The product stopped working automatically for me today but when I run the cron job manually it works just fine.

Any idea y2ksw?

[fxdigi-cash]

does this mod work??

I tried it, but the images are still linked externally from image uploaders...

is there something I'm missing here?

[sticky]

does this mod work??

I tried it, but the images are still linked externally from image uploaders...

is there something I'm missing here?

It does work. I've used it for years without issues.

Well, until now.

[fxdigi-cash]

It does work. I've used it for years without issues.

Well, until now.

Thanks for the feedback, but still in my case it is not working!!

I guess there is something wrong here. it is either the version I'm using (v1.0.5) or a misconfiguration !!

:confused:

[y2ksw]

Any idea y2ksw?

Did you have a look in the database? :)

[y2ksw]

does this mod work??

I tried it, but the images are still linked externally from image uploaders...

is there something I'm missing here?

If your server cannot access external servers or if external servers do not allow downloading images, they remain as is. Also, it may take some time to catch up.

[vietdotseo]

if i edit old posts, how new images in these posts ???

I have picture like https://lh5.googleusercontent.com/-meeECZ7DiCI/U9Mx7R0O4qI/AAAAAAAAA0Q/-wpqjsW6SMA/w738-h458-no/giao-an-lop-5-theo-chuong-trinh-moi.png

It don't Import

p/s: sorry my English

[sticky]

Did you have a look in the database? :)

Yes, but what am I looking for?

[sticky]

If your server cannot access external servers or if external servers do not allow downloading images, they remain as is. Also, it may take some time to catch up.

Is there only one hook? All I see is: Add and set iei_parsed field accordingly for eventual image parsing

I don't understand why when I click the cron manually it works but it just won't run automatically while all other crons work.

It's very weird.

[y2ksw]

Is there only one hook? All I see is: Add and set iei_parsed field accordingly for eventual image parsing

I don't understand why when I click the cron manually it works but it just won't run automatically while all other crons work.

It's very weird.

In fact. But it may be the product is disabled or it is scheduled in a way it never runs often enough.

The iei_parsed field is found in the post table and if it is 1 or more, it won't parse that post again.

[sticky]

Check that table and it said 1 so i set it to 0 on the posts it showed.

It still isn't working and I can't figure out what to do :(

[rustyshack]

I have had this mod installed on my one forum for years and it worked great.
Then I moved to a new server and upgraded to 4.2.2 VB
The new server has PHP 5.4 which i am thinking is the problem.

Before i discovered it wasn't working I installed this on another forum on the same brand new server, also running 4.2.2. On this forum it created a 2 folders and then stopped.
I cannot figure out what is stopping it, it seems it can't write files or something.

So i looked on my other forum and it stopped running the same day I moved it to my new server.
So there is either a problem with 4.2.2 or php 5.4
Cannot find any reason that neither will work and this is mainly the differences.

Even if I run it manually it doesn't work.

[sticky]

What would it explain it running manually but not automatically?

[Delfi_r]

Installed on 4.0x, I'm using it on 4.22pl1; php 5.3.2-1 and MySQL 5.1.63 and runs fine.

What I would ask is a version that works on blogs and their comments, too.

[AndrewSimm]

If I uninstall this can I delete the images in the imported folder? Will the original external link be back in the post?

[AndrewSimm]

I guess I know the answers to my questions. Is there a way to reverse this since the original url is stored in the DB?

[y2ksw]

I guess I know the answers to my questions. Is there a way to reverse this since the original url is stored in the DB?

It is possible to reverse, but it will need some crafting skills with queries.

I'm still unsure if you can do it with queries only, or if it requires also some scripting. If you would like to test if the source is still available, then scripts are definitively required.

[addamroy]

I have a few questions before I go ahead and use this. (And a big thanks for taking the time to make this)

1 - Does this mod have any lossy or lossless image compression features?

Would be nice if these images were compressed in some way during the import.

2 - If I install this mod, does it AUTOMATICALLY start importing images from past posts, or do I have to initiate that import manually via the mod options?

3 - Does this mod import images AS POSTS ARE MADE, or are they imported via a cron-job throughout the day?

And how often does the cron-job run?

4 - Why are there 2 maximum post count options? I notice there are 2 screenshots, each with similar options. Are both options pages accessible via the mod, or is each one a screenshot from v4 and v5 of this mod?

5 - I'm on a semi-shared server (urljet) and have somewhere around 200,000 posts. Will this mod be too server intensive for me or do I just need to lower the maximum post count options? Also, what post count per run do you recommend?

Thanks dude, this looks awesome.

[y2ksw]

I have a few questions before I go ahead and use this. (And a big thanks for taking the time to make this)

1 - Does this mod have any lossy or lossless image compression features?

Would be nice if these images were compressed in some way during the import.


2 - If I install this mod, does it AUTOMATICALLY start importing images from past posts, or do I have to initiate that import manually via the mod options?

3 - Does this mod import images AS POSTS ARE MADE, or are they imported via a cron-job throughout the day?

And how often does the cron-job run?

4 - Why are there 2 maximum post count options? I notice there are 2 screenshots, each with similar options. Are both options pages accessible via the mod, or is each one a screenshot from v4 and v5 of this mod?

5 - I'm on a semi-shared server (urljet) and have somewhere around 200,000 posts. Will this mod be too server intensive for me or do I just need to lower the maximum post count options? Also, what post count per run do you recommend?

Thanks dude, this looks awesome.
1. If you convert all images (option) then they become compressed with the jpg standard and also filtered against embedded scripts and other strange things.
2. Normally it starts alone.
3. cron; you can decide by changing the frequency
4. they are 2 screenshots depending on the mod version old/new. No vb5 tested yet but maybe it still works ...
5. It may be very server intensive, if you have a high parsing rate and a high number of posts at a single time. I would suggest to try this first on a small test environment. I also suggest to start with a small number of images. If you have an interval of 1 minute and 10 posts at each, it will catch up in about 15 days (1440 minutes x 10 x day). It depends a lot on how much your server can hold. On a professional dedicated server (4 core 3.4G) I have run this script with 1000 posts per minute with no noticeable problems.

[addamroy]

Thanks for the reply. Quick clarification though,

also filtered against embedded scripts and other strange things.

What do you mean by that?

[y2ksw]

Thanks for the reply. Quick clarification though,



What do you mean by that?

worms ;)

[addamroy]

worms ;)

Good because I hate worms. :)

[mservis]

Hello it's me not delete the images downloaded by the script, both the images and subdirectories marked out apache user and owner.

So do not let me change anything, or delete, or change attributes.

is there any solution?

[akxt660]

Hi!

This is an excellent product, thank you for offering us.

When this product will allow the user to switch between "imported images" and return to the "original links"? This is very important as it would be possible to save the images on the server and return to the external reference mode to save bandwidth and ensure the images for the future. Also, if there are problems with the operation or server, it would be possible to return to the original mode.

It would be important to allow: when return to the "imported images" mode, they are not reprocessed the rescue of those who are already on the server, only those that have not yet been redeemed (new links only and reactivate the internal reference).

I believe that after this update , this product would be perfectly safe and can even be used natively in vbulletin system.

Thank You!

[y2ksw]

Hi!

This is an excellent product, thank you for offering us.

When this product will allow the user to switch between "imported images" and return to the "original links"? This is very important as it would be possible to save the images on the server and return to the external reference mode to save bandwidth and ensure the images for the future. Also, if there are problems with the operation or server, it would be possible to return to the original mode.

It would be important to allow: when return to the "imported images" mode, they are not reprocessed the rescue of those who are already on the server, only those that have not yet been redeemed (new links only and reactivate the internal reference).

I believe that after this update , this product would be perfectly safe and can even be used natively in vbulletin system.

Thank You!
I do understand what you want, but it is not intended to reverse the process of imported images.

Normally, you would use this plugin to ensure to get all the images users have simply pasted and you will know beforehand the cost of the operation: bandwidth and disk space.

On very large sites (1.000+ active users), this will be about 1 GB/month or 1000 images.

[mservis]

No solution ?

https://vborg.vbsupport.ru/showpost.php?p=2523920&postcount=624

greetings

[y2ksw]

Hello it's me not delete the images downloaded by the script, both the images and subdirectories marked out apache user and owner.

So do not let me change anything, or delete, or change attributes.

is there any solution?

There is nothing we can do about it.

This happens if the server process has different access rights than your FTP access, and must be fixed by your provider.

[adwade]

I installed this on a VB 3.8 test forum and it appears to be working well. :up:

There should be no conflict at all, since it uses a different table and an additional field to distinguish parsed posts from new or modified ones. Just disable the 3.x one and install this ... you are ready to go. Please don't forget to enable the product once you have chmodded the imported file folder and setup the options, since it installs disabled.

I disabled my installed 3.x version (i.e. Image Cache v1.0.2.0 (https://vborg.vbsupport.ru/showthread.php?t=123338)), uploaded the new cron job, created a forum/images/imported directory, imported the product-iei4.xml (v1.0.2) file per the instructions. After toggling the Always Convert Images to Yes, I then enabled the product on vB3.8

I next edited and approved a moderated thread, after adding an external image to it and the image in the thread was subsequently processed correctly. However the MOD seems to have created some month subfolders (i.e. 03, 05, etc) that appear empty.

Q. I assume the MOD is working it's way through my oldest threads, but was just curious as to why some subfolders created have no contents?

[y2ksw]

R. In a few rare cases the procedure may not be able to terminate the import process timely, or encounter another error, and will leave then an empty folder. If you care about those, from the linux system crontab you may issue the following command once a month:
find /your/import/folder/ -type d -empty -delete
Please double-check the folder name ;)

[dorans]

regards,
How to download the original image size, not thumbnails?
With postimage.org or other ...

[y2ksw]

regards,
How to download the original image size, not thumbnails?
With postimage.org or other ...

Normally, the images are imported in their original size. If you intended to download a different, attached image to a specific URL, this is not possible.

[dorans]

Postimage via script sends the thumbnail link.
IE download icon instead of the whole picture ... Or I something wrong?

[y2ksw]

Postimage via script sends the thumbnail link.
IE download icon instead of the whole picture ... Or I something wrong?

The idea behind this plugin is to save the image of the URL the user has pasted into a post. Thus, if it is the thumbnail, the thumbnail is imported.

[dorans]

thank you,
I understand the function. The problem is for technical drawings. As thumbnails are useless.
Mod works completely correctly.

[y2ksw]

thank you,
I understand the function. The problem is for technical drawings. As thumbnails are useless.
Mod works completely correctly.

If there are reproducible rules between thumbnails and the drawings, then you may eventually replace the thumbnail URL's before running this plugin's task.

[dorans]

Me again :)

After the import I have this link:

https://vborg.vbsupport.ru/

How can I avoid this part? [URL = "http://postimg.org/image/txakyobrx/"] and [/ URL]
thanks,

[y2ksw]

Me again :)

After the import I have this link:

mysite.com/dddd/image/2014/11/108.jpg

How can I avoid this part? [URL = "http://postimg.org/image/txakyobrx/"] and [/ URL]
thanks,

You can't. This plugin does handles only image tags and ignores the rest. As the URL tag is surrounding the image, it is ignored. Eventually you may craft a special query which allows removing surrounding URL's, but this is far beyond of the scope here. (See MySQL REGEXP)

[dorans]

I assumed it.
The problem is that it remains a link to postimage. Gonna remove.
thanks,

[BadgerDog]

This is a great mod and it's been very useful to our forums over the past several years.. :up:

In fact, we sent y2ksw a large donation way back when to thank him for his work... ;)

The only serious problem we've ever experienced is that nagging issue of handling a picture someone uploads that's too large for the mod to handle.

Refer to old posts 167 and 168 (and older) where we discussed it and "Bird of Prey" also helped with figuring out what could be done, to some degree.

Since there wasn't an obvious solution then, we simply got into the habit of manually running the mod every morning during maintenance routines, to make sure it completed and didn't get hung on any large pics members had uploaded.

Well, that worked fine for the last 24 months and ironically, we never had one hang, except for this morning. A member must have uploaded an 8Mb pic sometime yesterday and today when we manually ran and checked completion of the mod, it simply hung at the very start. We turned on the logging to see if we could figure out where it was hung, but there was no data there, so it must have been the very first pic it tried to import at some point.

So, we began the laborious task of going through hundreds of threads for the last day or so, trying to find one with an IMG pic that was too large for the mod to handle. After a few hours we found the 8 Megabyte sized pic, downloaded and resized it, then put it back.

We then re-ran the import mod and it worked fine, completing the import of this (now smaller) pic, as well as others that had been uploaded in the last 24 hours.

So, why am I posting?

Well, it was a real pain in the a$$ to have to manually sift through all those threads looking for a single pic that cause the import mod to hang and lock up. So, I was wondering if anyone had any ideas how we could automate a search for excessively large pics, so we could manually fix them and get y2ksw's mod going again?

I gather you can't catch large pics before downloading them into the site, so it appears we have to live with the "hanging" issue, but it would be great if we could somehow find the offending pic(s) easier when that happens.

Anyone have any ideas?

Thanks..

Regards,
Doug

[y2ksw]

Well, one idea could be to attempt to download the image prior to examine its size, but then eventually you have other problems, such as streams, which could never finish.

[Johnny G]

You could always limit the size of a file to be uploaded to be <8Mb (or whatever the nominal size of the image is)?

[BadgerDog]

You could always limit the size of a file to be uploaded to be <8Mb (or whatever the nominal size of the image is)?

Hmm...

Where do I do that under vBulletin 4.13?

.. and how would vBulletin know the size of the file in advance of actually having to download it?

Thanks for the feedback ... appreciate it.. :)

Regards,
Doug

[y2ksw]

You could always limit the size of a file to be uploaded to be <8Mb (or whatever the nominal size of the image is)?

It sounds easy, but isn't.

It would require a raw socket connection and manual reading of the file, and stop reading if it has reached 8 MB. Then however, it would be possible to add a timer, too, and abort slow files as well. Also, it would allow to examine the file header before downloading the whole file and thus discard no-image files efficiently. But it would take a bunch of code and yet more options.

Then we remove the CURL code, but we'll loose the redirect features and SSL. Thus, it'll fail with Facebook, Imageshack, Google images, just to name a few.

I would say, it is a no-go.

[bmwfans]

Great idea! Do you know if this is going to work on 3.8.9?

[y2ksw]

Great idea! Do you know if this is going to work on 3.8.9?
This version here works for 4.x only, but you may try this:

https://vborg.vbsupport.ru/showthread.php?t=123338

[Hostboard]

How about an option to save to a cloud service such as OneDrive as a CDN and pull the images from there?

[y2ksw]

How about an option to save to a cloud service such as OneDrive as a CDN and pull the images from there?

You may use on Linux an external service as a mount for saving, for example curlftpfs or sshfs, and a redirect for showing those images. All other methods (such as using PHP ecc) go too far off the road, because there is never one system identical to another. I have had solutions like the above, but in the end I came up with Dropbox for some cases, and for all the rest, a larger local storage ;)

[majjed2008]

can we import the CMS image ? that it post inside articals there ?

Thank you

[hivitro]

Hi y2ksw, ty for this great mod, It has served me well these years.

I have a question before making a modification.

I am interested in taking a new .com domain and use images only, like a gallery.

My question is this. If I make a change in Forum URL replacement, and put a new domain, all new and old images will automatically move there?

Currently, my domain is http://www.nivelextremo.com the forum is inside / post / and the images are in / post / imgbackup /

If I for example, bought the domain like www.imgbackup.com or anything else and set

Forum URL replacement -> http://www.imgbackup.com/images

The old and new images, will move to the external domain? Or just new? or what would happen?


Thanks in advance

[y2ksw]

can we import the CMS image ? that it post inside articals there ?

Thank you

No, it works only for forum posts.

[y2ksw]

Hi y2ksw, ty for this great mod, It has served me well these years.

I have a question before making a modification.

I am interested in taking a new .com domain and use images only, like a gallery.

My question is this. If I make a change in Forum URL replacement, and put a new domain, all new and old images will automatically move there?

Currently, my domain is http://www.nivelextremo.com the forum is inside / post / and the images are in / post / imgbackup /

If I for example, bought the domain like www.imgbackup.com or anything else and set

Forum URL replacement -> http://www.imgbackup.com/images

The old and new images, will move to the external domain? Or just new? or what would happen?


Thanks in advance
Only the new images are moved. the old images can be moved, if you change the URL in all posts accordingly (with a REPLACE query on post.pagetext). The short term cheaper solution is a redirect, but it may add quite some load to your server, if you have a lot of images.

[hivitro]

Only the new images are moved. the old images can be moved, if you change the URL in all posts accordingly (with a REPLACE query on post.pagetext). The short term cheaper solution is a redirect, but it may add quite some load to your server, if you have a lot of images.

OK I understand. Time to work :D

Thank you.

Edit II > 1st Step Done:

1 - I move all images from inside my forum to another URL, i choose http://www.imageslocker.com and there put all from 2007 to 2015
2 - I Use Mysql UPDATE/REPLACE in post.pagetext and then, do al full Clear system cache.. table repairs, Forums Cache... and then all works ok.

Finally. New images, goes to the forum URL, the next year i will move in bulk again.

In the options IGNORE site i must write with/and not http and www, if not the mod copy again the images to the forum.

I just have a question.. I start to make some order/renames of files, in categories. How does this affect the table iei_img?

Becouse the old URL, will be the same, but the new URL it will not be because I moved / renamed.

[majjed2008]

Hello

after it import the image it dont show up in post I edited the post and i see the link point correctly to site./images/imported but the picture totally gone from all thread how to fix that please ?

Thank you

[hivitro]

Hello

after it import the image it dont show up in post I edited the post and i see the link point correctly to site./images/imported but the picture totally gone from all thread how to fix that please ?

Thank you


Firefox dont show your images if the URL is incorrect or the image is not where you think it is.

Use Chrome or IE to check. Make a test post, and link and external image. Go Administrator panel and run scheduled tasks > vB4 Import External Images

Go again to your test and check if it load the image imported or if show a broken avatar, do a right click in there and open in another windows, check the URL and you will see the error. Fix it in config and clear Cache in your navigator (CTRL+F5) and check again.

[majjed2008]

thank you very much i am going to do that , but i fined the the picture use wrong URL how to change it please ? do i have to thread and edited it manually i have very big forum more than 300K thread ?

[hivitro]

thank you very much i am going to do that , but i fined the the picture use wrong URL how to change it please ? do i have to thread and edited it manually i have very big forum more than 300K thread ?

Sure, once you run this mod, it will start to get a copy of all post/images inside your forum path.

Try to be sure when you make the config. IDK if the vbulletin cache, browser cache.. if you run the vb import, and you need to change anything, you need to clean all cache to be efective.

If you summit a mistake, like me, and write "galerry" and not, "gallery" you can fix with sql sentence inside of you db, iei_img table.

And remember to set a image for "noimage" pictures, its will reemplace a lost linked image if you want.

[y2ksw]

thank you very much i am going to do that , but i fined the the picture use wrong URL how to change it please ? do i have to thread and edited it manually i have very big forum more than 300K thread ?

If you have made a mistake, or simply moved the forum or the images to another place, you may use phpmyadmin and craft a query, like:

UPDATE post SET pagetext=REPLACE(pagetext, 'old', 'new')
e.g.:
UPDATE post SET pagetext=REPLACE(pagetext, 'http://www.abc.com/', 'http://www.def.net/')
or:
UPDATE post SET pagetext=REPLACE(pagetext, '/imported/', '/images/new-site/')
It is unavoidable to make first a backup of the involved table, then a few tests, and if everything is ok, issue the final query.

[Manoel J?nior]

If you have made a mistake, or simply moved the forum or the images to another place, you may use phpmyadmin and craft a query, like:

UPDATE post SET pagetext=REPLACE(pagetext, 'old', 'new')
e.g.:
UPDATE post SET pagetext=REPLACE(pagetext, 'http://www.abc.com/', 'http://www.def.net/')
or:
UPDATE post SET pagetext=REPLACE(pagetext, '/imported/', '/images/new-site/')
It is unavoidable to make first a backup of the involved table, then a few tests, and if everything is ok, issue the final query.
This may be something common, it could not be implemented in the product?

[y2ksw]

This may be something common, it could not be implemented in the product?

It is too dangerous to handle by ordinary forum owners. Who knows a little about MySQL eventually will do it with no harm, but a complete noob is capable to destroy the message base with one click. I can't be fund of such a problem.

[Manoel J?nior]

It is too dangerous to handle by ordinary forum owners. Who knows a little about MySQL eventually will do it with no harm, but a complete noob is capable to destroy the message base with one click. I can't be fund of such a problem.
You're right. Thank you

[mescalin]

Hello,

I installed thisplugin years ago and my imported folder size is very large and increasing. Is there a way to delete unlinked images from folder? Because many of the threats get deleted but the images remains there

[RichieBoy67]

Hello,

I installed thisplugin years ago and my imported folder size is very large and increasing. Is there a way to delete unlinked images from folder? Because many of the threats get deleted but the images remains there
Threats?:eek:

[Manoel J?nior]

Threats?
He meant threads

[hivitro]

Hello,

I installed thisplugin years ago and my imported folder size is very large and increasing. Is there a way to delete unlinked images from folder? Because many of the threats get deleted but the images remains there

They would be very good options for a future release.

[X] Delete unused backup images

[X] only save images of the 1st post, not answers

[y2ksw]

They would be very good options for a future release.

[X] Delete unused backup images

[X] only save images of the 1st post, not answers

The sense of this plugin is to keep all images and deleting threads etc is not a common practice, but only of a very few sites.

In order to efficiently delete unused images the only way is to use server side scripts and these may be crafted and used anyway only by system administrators ... there is no single system identical to another.

Ideally, you make a list of all images and search the posts, signatures and other places for those images. All of which you find, will be marked to keep. And then you delete the rest. Another approach would be to analyze the server logs over a long timespan (1 month or more) and mark all found images to keep.

[friendlymela]

This is good one

[y2ksw]

This is good one

I'm glad you like it :)

[friendlymela]

I'm glad you like it :)

would u like to share any screenshot of this mod?

[y2ksw]

would u like to share any screenshot of this mod?

There isn't any ... it's a background service. Besides the ACP options there is nothing to show ...

[final kaoss]

What if I want my images converted to png instead of jpg? I prefer using pnggauntlet on my images, plus I prefer png over jpeg anyday.

Also you may want to reword this. Do you mean the images are stored in a server cache and consuming ram or that the images are written out as files in the images/imported folder?

Cache image URL's into memory. If you have a large board, a lot of imported images and limited memory, you may want to switch this option off. If disabled, images are tested by less efficient database queries.

[y2ksw]

What if I want my images converted to png instead of jpg? I prefer using pnggauntlet on my images, plus I prefer png over jpeg anyday.

Also you may want to reword this. Do you mean the images are stored in a server cache and consuming ram or that the images are written out as files in the images/imported folder?

- You may change the code accordingly :)
- Neither. When importing images, the paths of images, which are already imported, may be stored in memory to quicken the import process, or be queried each time. If you have limited memory for your PHP process and a large quantity of imported images, you may want to switch the memory cache off. Just in case ;)

[Farsisat]

Hi
This product will be worked for me but a few month dos not work about 2/26/2015 until now

anybody there help me?

[Manoel J?nior]

Hi
This product will be worked for me but a few month dos not work about 2/26/2015 until now

anybody there help me?
when was the last time the CRON rolled?

[Farsisat]

when was the last time the CRON rolled?

Today
10:30 AM

but last night uninstall the plugin then reinstall this. not work!

[Dragonsys]

Today
10:30 AM

but last night uninstall the plugin then reinstall this. not work!

it ran at 10:30am, or will run next at 10:30am?

[Farsisat]

it ran at 10:30am, or will run next at 10:30am?

this morning the next time is (Today10:30 AM) but now says (Today09:20 PM)

i dont know what am i doing to work well

[Manoel J?nior]

Hi
This product will be worked for me but a few month dos not work about 2/26/2015 until now

anybody there help me?
This has happened to me, and I realize that there were corrupt images on my server, check in date order on your server the last image and delete it and turn the CRON in hand again and see if it worked.

[Manoel J?nior]

This has happened to me, and I realize that there were corrupt images on my server, check in date order on your server the last image and delete it and turn the CRON in hand again and see if it worked.
I wonder if my help was useful to you.

[mytreehouse]

This has happened to me, and I realize that there were corrupt images on my server, check in date order on your server the last image and delete it and turn the CRON in hand again and see if it worked.

I did the same. I deleted the last (corrupt) picture everytime the process stopped and started the task again, untill the process was completed.

[Manoel J?nior]

My temp folder is with many pictures with the final cached, I can remove all these files?

[Milobil]

Hi,

My forum has been created in 2013 and I installted the mod 2 days ago, however it has only imported the 2013 images in my server. And even though the script is run, it doesn't create anything else now. There are no 2014 or 2015 folders the my server..

Can someone help me please ?

[Milobil]

Hi,

My forum has been created in 2013 and I installted the mod 2 days ago, however it has only imported the 2013 images in my server. And even though the script is run, it doesn't create anything else now. There are no 2014 or 2015 folders the my server..

Can someone help me please ?

Any help please ?

[Dragonsys]

did you check the temp folder for bad images, as the others had issues with above?

[TerryMason]

I'd like to convert my site from http to https, and need to run this script to clear the way.

Is there a way to import the images using https, even though my forum url is using http only? Even better, can I have link them images using something like //www.forum.com/image.png, instead of specifying https.

*edit - never mind. I found that if I just put the desired url into the replament url section it works perfectly.

Thank you for this fantastic mod.

[Milobil]

did you check the temp folder for bad images, as the others had issues with above?

Where can i find this temp folder please ? Is it the same as "Imported" folder ?

Regards

[Milobil]

Well it seems it's going good now :o

[Kane@airrifle]

If I placed IMG="{param}" into the image tag option field would the import process retain image titles?

EG: [IMG="This is an image title"]http://etc.etc [/ img]

[y2ksw]

If I placed IMG="{param}" into the image tag option field would the import process retain image titles?

EG: [IMG="This is an image title"]http://etc.etc [/ img]
I can't remember ... why don't you try it? :)

[tommythejoat]

Unless you do something with the IMG tag, it does not accept a parameter and will not parse if one is present.

Several years ago, I created an ALT tag that uses the parameter as the image url and uses the argument as the contents of the alt attribute. I just changed that to use title instead of alt to conform with the current HTML recommendations.

[khalled]

did you check the temp folder for bad images, as the others had issues with above?

Where can i find this temp folder please

[khalled]

Originally Posted by Dragonsys View Post
did you check the temp folder for bad images, as the others had issues with above?
Where can i find this temp folder please
29 Oct 2016 12:48

I will be thankful if any one can help:confused:

[BadgerDog]

Hi Giovanni ....

We just migrated our server to a new hardware platform. The old one was 10 years old.

Unfortunately, the import mod we've been running for years has stopped working and it simply hangs when the cron job executes. (see attached)

I enabled the cron log to test, but there's nothing showing an information in the log for the Import External Images job, so I assume the script never even executes.

The new server is running Apache/2.4.18 (Ubuntu) with PHP Version 5.6.28, but I don't think that should be an issue.

I'm not even sure what to check to see why it's stopped working and just hanging on a manual run after all these years?

Any help would be appreciated...

Regards,
Doug

[Seven Skins]

May be the folder permissions needs to be fixed.
(read/write/create) rights (chmod 777)

[BadgerDog]

May be the folder permissions needs to be fixed. Change to 777

Thanks for suggestion and jumping in to help .. :)

I was just going to post a follow up...

Our developer got this fantastic mod working again .. :up:

Well, he really didn't do anything except as he says "added a bunch of new/updated image code development libraries to the server" ..

For whatever reason, that caused the mod to start working again immediately!!! :)

I'm so happy to have it running again.. our members would really miss it's ability to preserve image history from all the pics that they have built up and posted over these last many years.

Regards,
Doug

[y2ksw]

Hi Giovanni ....

We just migrated our server to a new hardware platform. The old one was 10 years old.

Unfortunately, the import mod we've been running for years has stopped working and it simply hangs when the cron job executes. (see attached)

I enabled the cron log to test, but there's nothing showing an information in the log for the Import External Images job, so I assume the script never even executes.

The new server is running Apache/2.4.18 (Ubuntu) with PHP Version 5.6.28, but I don't think that should be an issue.

I'm not even sure what to check to see why it's stopped working and just hanging on a manual run after all these years?

Any help would be appreciated...

Regards,
Doug
... and also maybe fopen() for URL's is not enabled, or perhaps you got once again memory issues? Is memory swapping early?

[y2ksw]

Thanks for suggestion and jumping in to help .. :)

I was just going to post a follow up...

Our developer got this fantastic mod working again .. :up:

Well, he really didn't do anything except as he says "added a bunch of new/updated image code development libraries to the server" ..

For whatever reason, that caused the mod to start working again immediately!!! :)

I'm so happy to have it running again.. our members would really miss it's ability to preserve image history from all the pics that they have built up and posted over these last many years.

Regards,
Doug
ok, I didn't see this :)

[BadgerDog]

ok, I didn't see this :)

No problem Giovanni .. :)

Running real well now.. again !!! :up:

Fabulous add-on to our site and much appreciated by our membership of retired military veterans. It does a great job of helping us gather and preserve historical photographs in a single location on our web site.

Regards,
Doug

[Ashlar217]

Perfect for helping me keep my site SSL.

Installed & working on 4.2.3. I ran the cron job in batches of 10, and no issues. Nice work :)

[RichieBoy67]

Been running this successfully on a few sites but today I am getting a ton of these database errors: I replaced the name of the clients domain.

Invalid SQL:
INSERT IGNORE INTO iei_img (
oldurl,
newurl
) VALUES (
'http://wtfurls.com/images/86.jpg',
'mysite.com'
);

[y2ksw]

Been running this successfully on a few sites but today I am getting a ton of these database errors: I replaced the name of the clients domain.

Invalid SQL:
INSERT IGNORE INTO iei_img (
oldurl,
newurl
) VALUES (
'http://wtfurls.com/images/86.jpg',
'mysite.com'
);
Can you please send me the whole error via PM? As it looks here, there is no problem at all.

[RichieBoy67]

Thanks

Database error in vBulletin 4.2.5 Beta 1:

Invalid SQL:
INSERT IGNORE INTO iei_img (
oldurl,
newurl
) VALUES (
'http://wtfurls.com/images/86.jpg',
'removed'
);

MySQL Error : MySQL server has gone away
Error Number : 2006
Request Date : Monday, January 30th 2017 @ 03:24:12 PM
Error Date : Monday, January 30th 2017 @ 03:26:13 PM
Script : removed/cronadmin.php?do=runcron&cronid=45
Referrer : removed/cronadmin.php?do=modify
IP Address : removed
Username : removed
Classname : vB_Database_MySQLi
MySQL Version :
Server : removed
User Agent :

[y2ksw]

MySQL Error : MySQL server has gone away
This is an issue with your database server, there is nothing I can do about it.

[RichieBoy67]

This is an issue with your database server, there is nothing I can do about it.
I wonder if the host blocked it.. was working fine before.

[y2ksw]

I wonder if the host blocked it.. was working fine before.
No, but they have changed the server settings in order to reduce the server load. Eventually you can get it back working by reducing the number of imported images for every run.

[RichieBoy67]

No, but they have changed the server settings in order to reduce the server load. Eventually you can get it back working by reducing the number of imported images for every run.
I only had it set to 1...

Anyway we can get this to import Tapatalk images?

[KevinL]

Figured it out. Great mod.

Thanks!

[KevinL]

Question, I am having an issue where a few images are being replaced with different images from the source image. Source images and replacement images are all different in different posts/threads.

Any insight on this issue? Let me know what you need from me.

Thanks

[y2ksw]

I only had it set to 1...

Anyway we can get this to import Tapatalk images?
In theory yes, however I'm unsure about the technical problems.

[y2ksw]

Question, I am having an issue where a few images are being replaced with different images from the source image. Source images and replacement images are all different in different posts/threads.

Any insight on this issue? Let me know what you need from me.

Thanks
No idea. Unless the server is feeding random images ...

[Kane@airrifle]

I only had it set to 1...

Anyway we can get this to import Tapatalk images?

In theory yes, however I'm unsure about the technical problems.

Works just fine on my site.

[RichieBoy67]

Works just fine on my site.
How did you accomplish it?

[Kane@airrifle]

How did you accomplish it?

Not sure what you mean? Any external image url that sits between [img] tags in the post table is imported as is intended by this mod. Certainly is the case on my board unless I missing some other Tapatalk function..

[djbaxter]

Version 1.0.4 is temporarily in BETA stage until the users have confirmed the newly requested features.

Version 1.0.5 is still in BETA stage. It adds a new bunch of additional features, one of which allows you to test your mathematical understanding ... and a few other to fine-tune CURL and socket timeouts, and resizing on the fly of too large images. If you don't know what these options do, please leave them at their default values.

Are these two versions still in beta?

[djbaxter]

I decided to install version 1.05 which seems to be working generally quite well except for the following error:

Database error in vBulletin 4.2.3:

Invalid SQL:
INSERT IGNORE INTO vbiei_img (
oldurl,
newurl
) VALUES (
'http://www.websmileys.com/sm/love/458.gif',
'https://MYSITE/clear.gif'
);

MySQL Error : MySQL server has gone away
Error Number : 2006
Request Date : Monday, February 20th 2017 @ 12:24:53 AM
Error Date : Monday, February 20th 2017 @ 12:26:54 AM
Script : http://MYSITE/cron.php?rand=1487550292
Referrer : https://MYSITE/showthread.php?{VARIOUS THREADS}
IP Address : {VARIOUS}
Username : Unregistered
Classname : vB_Database_MySQLi
MySQL Version : {BLANK but is actually 5.6 / 10.0.29-MariaDB}


What's interesting is that so far at least it is always the same image (see above) but different threads and your add-on is in fact correctly inserting clear.gif in those threads.

I'm on a tweaked dedicated server so I don't think the "MySQL server has gone away" is the usual problem of too many connections (persistent connections is off)... not sure what to make of this.

Any suggestions?

Is it a bad idea to be running version 1.05? (I didn't get a response to my previous question above.)

[BadgerDog]

I'll add my 2 cents.... :)

I've been using y2ksw's mod for years and it's been working flawlessly on our v4.1.3 pl8 ... :up:

I check every morning by manually running his "vB4 Import External Images" cron job, just to see if it's hanging on any image imports, which way back in early versions used to be the result of it trying to import some HUGE freakin image a member had linked to in a post.

I haven't had it hang up for years, but I still check it as an old habit.... ;)

I'm running v1.06, but it's been so long, I can't remember why. That's what happens when you get to be almost 70. :D

I know I had PayPal donated to y2ksw for his excellent work and I also think I did more PayPal's a few times more as he privately helped me with some issues. I may have ended up with v1.06 as a result. For some reason, I think along the way working with him, he did something to ensure that it didn't crash when some clown member tries to link to a picture off-site that's too large for the code to handle. I think he also had me change some PHP values, but again, it's been so long since I've had to touch anything.

v1.06 is what my AdminCP currently shows me running and it's been reliable and probably one of the best vBulletin add-ons we use.

It's amazing to see images from members that go back years and years, which have long since been deleted from their original off-site linked to storage, still be showing in their posts. Being a research site, it has been extremely helpful in keeping old threads with pictures still relevant after several years, instead of getting that dreaded "Photo Missing" error.

Thanks again for your excellent add-on y2ksw ... :up:

Hope that helps someone else here ... :)

Regards,
Doug

[RichieBoy67]

I would be glad to donate towards an updated version that addresses the issues I mentioned in my above post.

[Sonik_elf]

Hello.
Sorry.. I installed it .. And? How to start the scanning process and import images?

[djbaxter]

1. Go to AdminCP >> Options >> Settings >>vB4 Import External Images and input your settings and enable the addon

2. Go to AdminCP >> Scheduled Tasks >> Scheduled Task Manager >> vB4 Import External Images

3. Edit your settings there and then enable the cron job /scheduled task.

[djbaxter]

It's now occurring with other images:

Invalid SQL:
INSERT IGNORE INTO vbiei_img (
oldurl,
newurl
) VALUES (
'http://blog.lib.umn.edu/shea0017/philosophy/MATISSE%20ICARUS.JPG',

...

MySQL Error : MySQL server has gone away


Missing images seem to create a problem?

[Sonik_elf]

1. Go to AdminCP >> Options >> Settings >>vB4 Import External Images and input your settings and enable the addon

2. Go to AdminCP >> Scheduled Tasks >> Scheduled Task Manager >> vB4 Import External Images

3. Edit your settings there and then enable the cron job /scheduled task.

It performs the task, but do not appear in the image folder. What is the problem? Folder have 777 permissions.

Can somewhere view logs or something?

[djbaxter]

Only missing images appear in the image folder.

[djbaxter]

Update:

It appears that the error messages I received and posted above were misleading (at least they misled me). It is not the case that this add-on is finding the same missing image in a number of posts; I thought that initially made sense because the image pointed to a no longer available smiley.

Rather it seems that once the add-on encounters a no longer available image, it gets "stuck" on that image each time it tries to process another run, and keeps generating the same error.

A work-around which appears to be working for me is to add the URL (or part of the URL) pointing to the problem image to the add-on's Ignore list:

AdminCP >> Settings >> Options >> vB4 Import External Images >> Ignore

Ignore
Images containing these values are ignored and will not be replaced. Normally you will enter here your media server or gallery folder. Images of your forum and the imported folder are always ignored. Please separate each item by a new line.

My ignore list now looks like this:


imgcache/
websmileys.com/
/shea0017/
allposters.com/


Now that I'm posting this, I'm not sure what "imgcache/" is - that was prepopulated by the add-on.

[y2ksw]

MySQL server has gone away
Nothing we can do about. Server issues are server issues.

Now that I'm posting this, I'm not sure what "imgcache/" is - that was prepopulated by the add-on.
It's the folder where imported images are stored. Values in this field avoid to import images which were already imported or stored elsewhere by other means.

[djbaxter]

I think you've missed the point. See my latest post above. The add-on gets stuck on a missing image and keeps trying to retrieve it even after it moves on to other posts.

https://vborg.vbsupport.ru/showpost.php?p=2582747&postcount=723

[y2ksw]

I think you've missed the point. See my latest post above. The add-on gets stuck on a missing image and keeps trying to retrieve it even after it moves on to other posts.

https://vborg.vbsupport.ru/showpost.php?p=2582747&postcount=723
We have had stuck image problems in the past, namely out of memory problems with huge images and PHP silently crashing, but then, MySQL would not complain. None of the stuck image problems were solvable by more clever PHP code. And trust me, we have tried hard (a customer paid all the effort but no avail).

All I can say is, when you cannot solve the timeout problems, the plugin does not work for you.

[xyd]

I'm actually strongly considering developing a service for forum owners, as I am one myself, to do what this plugin did (or does, I don't know if it still works) along with a few extra features... all things I've been doing on my forum for a few years but utilizing custom code and not vB plugins...

There's almost nothing I hate more than coming to a forum and reading quality posts from a while back but the images not working in them.

What I've been doing, and the service that I'm considering starting up, would do these things:

1.) Any external images in the past that have been posted to your forum, which are still available, would be imported into my server cluster.

2.) After being imported, the image will be optimized (using lossless - or lossy - the site owner would be able to choose which they would want) using the latest image optimization routines (OptiPNG, etc etc etc).

3.) After being optimized, the old external images inserted into posts on your forums would be updated to no longer route to the external image, but the one on my server.

4.) Files would be distributed to your viewers over our CDN to allow for a better (by not having to wait for things to load) browsing experience.

5.) Any future posts made with external images inserted into them would have them automatically imported, optimized, posts updated and distributed through our service automatically.

The setup would be extremely painless, it'd pretty much be a set it and forget it type thing - everything would be done in the background.

I have a lot of resources at my disposal, but I'd charge around $10/month for access to this service (you can enable it for all of your forums - if you run multiple forums) because there are infrastructure costs if I were to provide this as a service.

Also you'd have the option to have all your normal image uploads (i.e. when someone attaches a jpg to a thread they are making) served through our service as well - the idea here would be to reduce strain on your server, reduce your bandwidth costs and improve browsing experience by faster page loading thanks to the files being served via CDN.

Another thing, and this is still experimental (but I'm implementing it on my forum as we speak), you would have the option to allow us to optimize your images even further, by making it so images that are way too big for the viewer's screen would be resized automatically during the import->optimization phase, into 4 different files (based on the most popular browser screen resolutions), which would not only increase their page loading speed, but decrease their bandwidth used and decrease your bandwidth used. They would, of course, be able to see the image in all it's glory by clicking on it, but the initial view would be sized intelligently (responsive web design). Images would never be upscaled - QUALITY and performance is something I require when it comes to processing images, so you would never see crappy quality images due to our service.

LASTLY, lol, this is still experimental as well (another thing I'm implementing on my forum as we speak), but images could be served as BPGs instead of JPGs/PNGs - more or less this would decrease the file size of images by about 30-40%, saving you and your viewers bandwidth, and making pages load even faster. Should this option be offered, and the forum owner want it enabled, you'd have to stick a piece of javascript into your forum (would all be done via the vB plugin) since BPGs aren't natively supported by browsers yet, but it's well worth doing. I'll post some before and after screenshots once I'm done implementing everything on my own forum.

As I said before, this would be an extremely powerful tool for any forum, there would be hardly any work required to implement it on your own forum. And of course since it would be provided as a service, I'd help anyone who needs it when it comes to installation/configuration.

I'm curious to see what kind of interest a service like this would conjure up - I know if it was around when I created my forum I would have definitely jumped on it for $10/month.

[y2ksw]

I'm actually strongly considering developing a service for forum owners, as I am one myself, to do what this plugin did (or does, I don't know if it still works) along with a few extra features... all things I've been doing on my forum for a few years but utilizing custom code and not vB plugins...
(...)
Sounds challenging :)

[BadgerDog]

I'm curious to see what kind of interest a service like this would conjure up - I know if it was around when I created my forum I would have definitely jumped on it for $10/month.

Nice initiative, but at $120/year, it sounds expensive when this mod does a similar job for the single one time price of a PayPal donation?

If you went ahead, it would also have to be an SSL site..

Regards,
Doug

[y2ksw]

Nice initiative, but at $120/year, it sounds expensive when this mod does a similar job for the single one time price of a PayPal donation?

If you went ahead, it would also have to be an SSL site..

Regards,
Doug
It surely depends on the service such an add-on may give. Say it is perfect and never crashes, nor has any kind of problems, ever, then $100/yr is not cheap, but in relation to the effort.

The point is that high end servers have a cost which never is paid unless some tricks apply. For example, I was playing with the idea of using flickr and instagram for storing images, taking advantage of a paid account. Then, the service would handle the downloading and redirecting only. But still, a professional service is very costly both in time and hardware.

Personally I prefer a low-end solution and no or little costs, at the expense of sometimes not getting exactly what I want.

[xyd]

I'll add a little more in response later today as I only have a few seconds at this very moment, but I wanted to mention something right now at least, regarding this:

Nice initiative, but at $120/year, it sounds expensive when this mod does a similar job for the single one time price of a PayPal donation?

If you went ahead, it would also have to be an SSL site..

Regards,
Doug

I agree, the service wouldn't be for everyone - a lot of people like self hosting (I myself am one of those people) - and perhaps there would be a self-hosted version I'd make available, but to provide the level of service I want to with such a mod/plugin/whatever you want to call it, being able to fully administrate the infrastructure is something that I see, personally, as being important to me as the developer.

In regards to it being ran on HTTPS, I entirely agree. In fact, it's something I strongly believe in so much that I wouldn't even give the option to disable SSL for the service. Some could argue that having images behind SSL could cause extra round trip time due to SSL negotiation, but these images being served from a CDN would clear that up for most forum operators - and in a lot of situations, even being behind SSL but served with a CDN images would still load faster due to not a lot of places using CDNs for forum images/attachments.

That, along with all of the other newer technologies that increase loading time (HTTP/2 etc), would all be enabled, and again, such extreme performance optimizations aren't as common on forums as we'd all prefer. I am extremely... paranoid, or obsessive... when it comes to optimization when it comes to delivery of web objects.

SSL is becoming a very important aspect to the web these days, and I'd sweeten the pot for those who would use this service if I do end up creating it (I'm looking to create a few services for forum owners actually, this one included)... by allowing the site owner to choose the subdomain they wish to have their images served from, and thus creating, utilizing and issuing (yes, giving you the actual cert to utilize on your site however you want) a wildcard SSL certificate (signed by GlobalSign's AlphaSSL CA) for your domain for free.

Will add a little bit more later when I get back.

[BadgerDog]

I agree, the service wouldn't be for everyone - a lot of people like self hosting (I myself am one of those people) - and perhaps there would be a self-hosted version I'd make available, but to provide the level of service I want to with such a mod/plugin/whatever you want to call it, being able to fully administrate the infrastructure is something that I see, personally, as being important to me as the developer.


Paying one time for a functional commercial product that would enable self hosting would be something we'd be interested in .. :up:

Thanks .. :)

Regards,
Doug

[RichieBoy67]

Ok, well this thread is for this mod..

Can anyone tell me why Tapatalk images do not import. I believe they used to but I am not sure. It could have to do with their https:// url change or ours but I really want those images on our server..

[djbaxter]

Why would the cron job for this task suddenly stop logging events?

[y2ksw]

Why would the cron job for this task suddenly stop logging events?
If PHP silently crashes, for example.

[y2ksw]

Ok, well this thread is for this mod..

Can anyone tell me why Tapatalk images do not import. I believe they used to but I am not sure. It could have to do with their https:// url change or ours but I really want those images on our server..
Maybe ... you have no CURL? fopen() is somewhat limited with https.

[djbaxter]

Why would the cron job for this task suddenly stop logging events?

If PHP silently crashes, for example.

I'm not sure what you mean by "silently crashes". I haven't seen anything to suggest anything crashed in server alerts. But assuming this might happen, what is the resolution? Server reboot?

[y2ksw]

I'm not sure what you mean by "silently crashes". I haven't seen anything to suggest anything crashed in server alerts. But assuming this might happen, what is the resolution? Server reboot?
If PHP crashes silently then there is no notification anywhere and the causes are unknown. We have had silent crashes with huge images (such as 60.000x80.000 pixel). The solution is to mark that post as "scanned ok" and eventually exclude the specific image in the options, if it appears in more than one post.

[djbaxter]

If PHP crashes silently then there is no notification anywhere and the causes are unknown. We have had silent crashes with huge images (such as 60.000x80.000 pixel). The solution is to mark that post as "scanned ok" and eventually exclude the specific image in the options, if it appears in more than one post.

How do you mark it as "scanned ok"? How do you even know which image to do that for if there's no notification and if the add-on is no longer logging its activity?

[RichieBoy67]

I just checked and this does not appear to be working at all any more.. no changes to the settings or plug in but updated Vbulletin, changed php version..things like that. Vbulletin 4.2.4 beta 1..also changed site to https://

Cron tasks says done but nothing happens..

[RichieBoy67]

Just found this in my log:

Database error in vBulletin 4.2.5 Beta 1:

Invalid SQL:
INSERT INTO oasl_session SET userid='85961', session_token='JEOGPb3rGt1nfI7PfB2LXGTq8B6xJazzQGD v', date_created='1488993068';

MySQL Error : Duplicate entry '85961' for key 'userid'
Error Number : 1062
Request Date : Wednesday, March 8th 2017 @ 12:11:07 PM
Error Date : Wednesday, March 8th 2017 @ 12:11:08 PM
Script : http://##############
Referrer : http://#######
IP Address : 173.245.50.139
Username : #######
Classname : vB_Database_MySQLi
MySQL Version :
Server : ########

It is supposed to be https://// also not sure about the duplicate session..

[djbaxter]

@RichieBoy67: That's essentially what is / was happening for me.

I recently updated to vBulletin 4.2.4 but I was getting all that with 4.2.3 PL2.

[y2ksw]

How do you mark it as "scanned ok"? How do you even know which image to do that for if there's no notification and if the add-on is no longer logging its activity?
The posts are scanned top-down (from postid=1 to latest).

The plugin adds a field to post, called iei_parsed.

If you issue the query:

SELECT postid
FROM post
WHERE iei_parsed>0
ORDER BY postid DESC
LIMIT 1you get the last post ID which was successfully parsed. If this value stays put, then something is not working.

As for the concerns with https and SSL (I've been looking into it now), requesting data from such sites requires you to use CURL. You can choose the mode from the options. CURL is installed anyway since required by 4.x vBulletin versions.

Marking a post "OK" can be done this way:

UPDATE post SET
iei_parsed=1
WHERE postid=Nwhere N is the postid.

[y2ksw]

Just found this in my log:

Database error in vBulletin 4.2.5 Beta 1:

Invalid SQL:
INSERT INTO oasl_session SET userid='85961', session_token='JEOGPb3rGt1nfI7PfB2LXGTq8B6xJazzQGD v', date_created='1488993068';

MySQL Error : Duplicate entry '85961' for key 'userid'
Error Number : 1062
Request Date : Wednesday, March 8th 2017 @ 12:11:07 PM
Error Date : Wednesday, March 8th 2017 @ 12:11:08 PM
Script : http://##############
Referrer : http://#######
IP Address : 173.245.50.139
Username : #######
Classname : vB_Database_MySQLi
MySQL Version :
Server : ########

It is supposed to be https://// also not sure about the duplicate session..
These are crashed database issues and require a table repair.

[y2ksw]

I just checked and this does not appear to be working at all any more.. no changes to the settings or plug in but updated Vbulletin, changed php version..things like that. Vbulletin 4.2.4 beta 1..also changed site to https://

Cron tasks says done but nothing happens..
Please disregard my PM and see here:

https://vborg.vbsupport.ru/showpost.php?p=2583517&postcount=743

[djbaxter]

The posts are scanned top-down (from postid=1 to latest).

The plugin adds a field to post, called iei_parsed.

If you issue the query:

SELECT postid
FROM post
WHERE iei_parsed>0
ORDER BY postid DESC
LIMIT 1you get the last post ID which was successfully parsed. If this value stays put, then something is not working.

As for the concerns with https and SSL (I've been looking into it now), requesting data from such sites requires you to use CURL. You can choose the mode from the options. CURL is installed anyway since required by 4.x vBulletin versions.

Marking a post "OK" can be done this way:

UPDATE post SET
iei_parsed=1
WHERE postid=Nwhere N is the postid.

Thank you.

Running the first query shows 242442 as the last post ID.

Manually running the cron job leaves that number unchanged.

Running the second query yields

Affected Rows: 0 (0.0015s)

Repeating the whole sequence gives the same result.

Does that mean the add-on has stopped working?

Note my statement above that I recently updated to 4.2.4 but the issues were happening prior to that update.

[RichieBoy67]

The posts are scanned top-down (from postid=1 to latest).

The plugin adds a field to post, called iei_parsed.

If you issue the query:

SELECT postid
FROM post
WHERE iei_parsed>0
ORDER BY postid DESC
LIMIT 1you get the last post ID which was successfully parsed. If this value stays put, then something is not working.

As for the concerns with https and SSL (I've been looking into it now), requesting data from such sites requires you to use CURL. You can choose the mode from the options. CURL is installed anyway since required by 4.x vBulletin versions.

Marking a post "OK" can be done this way:

UPDATE post SET
iei_parsed=1
WHERE postid=Nwhere N is the postid.When I run this I get nothing..

##I get an id now after running a table repair. :) I will run the other one ..

[RichieBoy67]

I think it's working now because I do see the id moving after I run the 1st query. I do not see any new images yet in my import folder but I assume that just means it has not found any to import yet?

Thanks for your help!

[djbaxter]

Still not working for me. I did a repair and optimize on the database. No errors found.

[RichieBoy67]

Nope, not working.. looks like it went through the posts and is caught up with the last post but it has not imported any of the images..