vb.org Archive - Administrative and Maintenance Tools

Page 4 of 22

Show 40 post(s) from this thread on one page

vb.org Archive (https://vborg.vbsupport.ru/index.php)

- vBulletin 4.x Add-ons (https://vborg.vbsupport.ru/forumdisplay.php?f=245)

- - Administrative and Maintenance Tools - Import External Images (https://vborg.vbsupport.ru/showthread.php?t=253309)

BirdOPrey5

01-24-2011 09:23 PM

Quote:

Originally Posted by BadgerDog (Post 2154004)

Wouldn't it be better to process the oldest threads first, simply because users who fail to renew their off-site storage, or their free 30 day trials run out, would get those pics moved internally first so there's less chance of ending up with missing pics when that happens? :confused:

By the way, thanks for the tips .. :)

Regards,
Doug

The way I looked at it were older threads probably already are missing images so they don't matter as much... I'm sure it can be debated either way.

jimsflies

01-25-2011 01:18 AM

I've gone back and forth with this issue (photo links being broken) and using this mod. How hard is it, if a member asks to have his or her photos removed to remove them from the server? I think the members still own the copyright to their photos and if they wish to pull it, it ultimately is their prerogative, right?

BirdOPrey5

01-25-2011 02:01 AM

In my terms of service (and just about every forum TOS I ever saw) the simple act of posting anything on your forum be it text or picture grants you an irrevocable lifetime license to do with the content as you please. If the user owns the copyright they grant you a license to display it forever. The issue arises when a user posts an image they don't own the copyright to... in that case what is supposed to happen is the copyright holder sends you a take down notice you must comply with or face a lawsuit in which case you should comply.

However there is one well known company skipping the take down notice and going right to the lawsuits so it is a tough call.

y2ksw

01-25-2011 07:13 AM

Quote:

Originally Posted by BirdOPrey5 (Post 2153975)

I want to suggest one more change I've used successfully on my test forum that may be of use to people with established forums.

In that query change ORDER BY postid to ORDER BY postid DESC so that the newest threads are processed first since it might take days or weeks to process a large existing forum.

So the current option requests are:

make it possible to include and/or exclude forum sections
change the order

y2ksw

01-25-2011 07:19 AM

Quote:

Originally Posted by jimsflies (Post 2154154)

I think it would be very difficult to accomplish this, because how would you handle quotes?

If I do delete posts (rarely), I hide the whole thread the author has started, but will not touch any of quoted and otherwise copied posts. Including photos. As for the Terms of Service (TOS), I never will delete any post. It has been saving me several times from getting sued by forum fighters ;)

BirdOPrey5

01-25-2011 12:44 PM

y2ksw, if you're taking feature requests, here is some code I made to keep the original file names...

1) It takes only the file name, not the path-
2) It strips all non alpha-numeric characters besides underscores
3) It sets a max length and truncates any characters over the max length
4) It still appends an integer number to the end so there will never be any file name conflicts.

I added 1 new function replacing iei_getfile_index with:

PHP Code:


		
			
//Custom Code To Save Filename
function iei_get_file_index_name(&$path, &$extension, $oldfilename)
{
   //Get only the file name (no directory info)
   $oldfilename = basename ($oldfilename, $extension);
   //Strip out non alpha-numeric characters
   $oldfilename = preg_replace("#[^A-Za-z0-9_]#", "", $oldfilename );
   //Set max file name length to 40 characters
   $oldfilename = substr($oldfilename, 0, 40);
   
   //Increment number if file exists
    for($i = 1;; $i++)
    {
        $filename = "$path/$oldfilename-$i.$extension";
        if(!file_exists($filename))
        {
            return $filename;
        }
    }
}

And then the call to the function under the //Get File Index comment I changed it to:

PHP Code:


		
			
  $filename = iei_get_file_index_name($path, $extension, $value);

And it's working great... brings in the files with their own (clean) names... In the off chance the file has the same name as another one, it increments the counter by 1 so they always save with a unique name.

Feel free to use some, all, or none of this as you see fit. :up:

BadgerDog

01-26-2011 11:57 AM

Quote:

Originally Posted by BirdOPrey5 (Post 2154316)

y2ksw, if you're taking feature requests, here is some code I made to keep the original file names...

I'm an old Fortran programmer out of the 60's, so in order to save us the potential of screwing up the existing PHP file patching it, is there any chance of getting a copy of yours for testing with those changes already made? Thanks .. :)

Second, any idea how long you think it would take to run a 150,000 post forum through this process at 10 posts per shot?

Regards,
Doug

210665

01-26-2011 12:50 PM

Quote:

If it does not work at all, probably your server cannot use sockets, CURL or is locked out for retrieving web content by a firewall.

PS: Have you enabled the product after installing?

Yes I have activated after the installation
curl but I really should be enabled

y2ksw

01-26-2011 08:34 PM

Quote:

Originally Posted by BadgerDog (Post 2154826)

I'm an old Fortran programmer out of the 60's, so in order to save us the potential of screwing up the existing PHP file patching it, is there any chance of getting a copy of yours for testing with those changes already made? Thanks .. :)

Second, any idea how long you think it would take to run a 150,000 post forum through this process at 10 posts per shot?

Regards,
Doug

This depends on the posts with images, external servers, script timeout, computing power of servers. On one of my customers site with a top class server, it took 1 week to catch up the latest post, out of about 900.000 posts with about 15% posts with images and a cronjob every one minute (all *).

OldSchoolDSL

01-27-2011 12:28 AM

Question:

Wouldn't this eat bandwidth? I mean technically speaking, the more photos on your site that have to load... The slower things become, right?

I'm sure this helps in some way... I just lack the vision at this moment to see it.

Not to add the idea that it would add to your file system?

BirdOPrey5

01-27-2011 01:29 AM

It would use more of your own bandwidth but you stop "stealing" other people's bandwidth and you have total control over the images so if someone decides to delete or change them one day you still have them.

I know I have a user who posts a weather map graphic from time to time. It gets updated every few minutes, so if I do end up using this on my live forum I should remember to put whatever domain it's from on the ignore list else the image will never update.

OldSchoolDSL

01-27-2011 03:16 AM

Maybe I'm evil... But I'd rather eat someone else's bandwidth / server load time.

If I could find a cheap (yet fast) way of loading images on another server... I think I would.

SoloX

01-27-2011 07:11 AM

Not sure if this has been said before but if you are installing this mod, please be careful about copyrights. Inline image links *might* save you a court date but actually downloading and serving these from your server will def. get you in trouble.

google "righthaven lawsuits" and you will know why I am posting this caution.

sticky

01-27-2011 09:49 AM

Quote:

Originally Posted by BirdOPrey5 (Post 2154316)

PHP Code:


		
			
//Custom Code To Save Filename

function iei_get_file_index_name(&$path, &$extension, $oldfilename)

{

   //Get only the file name (no directory info)

   $oldfilename = basename ($oldfilename, $extension);

   //Strip out non alpha-numeric characters

   $oldfilename = preg_replace("#[^A-Za-z0-9_]#", "", $oldfilename );

   //Set max file name length to 40 characters

   $oldfilename = substr($oldfilename, 0, 40);

   

   //Increment number if file exists

    for($i = 1;; $i++)

    {

        $filename = "$path/$oldfilename-$i.$extension";

        if(!file_exists($filename))

        {

            return $filename;

        }

    }

}

And then the call to the function under the //Get File Index comment I changed it to:

PHP Code:


		
			
  $filename = iei_get_file_index_name($path, $extension, $value);

Ok, I'm going to try to install is.

So iei_getfile_index I replace with all the code you posted?

As for the last part, sorry what am I replacing exactly? What is the call to function?

vietfancy

01-27-2011 09:58 AM

thanks for a great mod.

How do I add more tags into this mod? Right now we can only use [img] tag. I do have [IMGL] and [IMGR] on my forum to align images to the left or to the right of the post.

Thanks again for a great mod.

y2ksw

01-27-2011 10:01 AM

Quote:

Originally Posted by SoloX (Post 2155163)

Normally you would have a TOS where you don't assume any responsibility for pasted images by your users.

As for the copyright, you don't steal images by copying them into your site. Google Images would do a great damage to all the sites they copy images from, in order to show them to everybody, don't you agree? You steal them from the moment you are using them improperly. For this reason I am a bit concerned about importing images from CMS posts, since the authors are usually administrators and thus should not steal images from other sites to make their own site nice and interesting ;)

y2ksw

01-27-2011 10:03 AM

Quote:

Originally Posted by vietfancy (Post 2155202)

It's a good moment to ask for new features, but I'm not sure if I implement them all :)

BirdOPrey5

01-27-2011 12:14 PM

Quote:

Originally Posted by sticky (Post 2155199)

Ok, I'm going to try to install is.

So iei_getfile_index I replace with all the code you posted?

As for the last part, sorry what am I replacing exactly? What is the call to function?

You could replace the old function with mine, or you could keep the old function and just add my function to the code.

As for the "call to the function" it's the line directly under the comment
//Get File Index

Replace the line:

PHP Code:


		
			
  $filename = iei_get_file_index($path, $extension);

With

PHP Code:


		
			
  $filename = iei_get_file_index_name($path, $extension, $value);

Or better yet just comment out the first line and add the other one below it so you have the original code if you ever want to go back:

PHP Code:


		
			
  //$filename = iei_get_file_index($path, $extension); 

  $filename = iei_get_file_index_name($path, $extension, $value);

BadgerDog

01-27-2011 06:26 PM

Quote:

Originally Posted by BirdOPrey5 (Post 2154316)

y2ksw, if you're taking feature requests, here is some code I made to keep the original file names...

I wanted to test your PHP version against just a specific thread to see how it worked, so I added something y2ksw showed me in a previous post to his code.

$SQL = "SELECT postid, " . TABLE_PREFIX . "post.dateline, pagetext, forumid
FROM " . TABLE_PREFIX . "post join " . TABLE_PREFIX . "thread on (" . TABLE_PREFIX . "post.threadid = " . TABLE_PREFIX . "thread.threadid)
WHERE iei_parsed = 0
AND threadid=28420
AND pagetext LIKE '%[/IMG]%'
ORDER BY postid";

When I did that with your PHP file, I get an SQL error, while the same piece of code addition in y2ksw's code executed correctly ONLY against that thread ID? :confused:

I like the idea of keeping the original filenames (or a variant thereof) and thank you for showing us that method, but I wanted to ensure it would work cleanly on our site before I turned it ON live against our whole forum.

Ami I missing something? Sorry, I'm not a programmer ... :D

Regards,
Doug

sticky

01-27-2011 07:09 PM

Quote:

Originally Posted by BirdOPrey5 (Post 2155239)

PHP Code:


		
			
  $filename = iei_get_file_index($path, $extension);

With

PHP Code:


		
			
  $filename = iei_get_file_index_name($path, $extension, $value);

Or better yet just comment out the first line and add the other one below it so you have the original code if you ever want to go back:

PHP Code:


		
			
  //$filename = iei_get_file_index($path, $extension); 

  $filename = iei_get_file_index_name($path, $extension, $value);

Working great, thank you!

BirdOPrey5

01-27-2011 08:16 PM

Quote:

Originally Posted by BadgerDog (Post 2155365)

My SQL query was altered a little so I could disable it by forum if I wanted (I added a JOIN statement), the old query was still there commented out- you could try putting the original query back in it and see what happens.

But this is my last post on this issue- this is not my mod, I'm not supporting any changes... I released my code so someone who knew what they were doing could implement it at their risk if they chose. I'm not going to pollute this mod's thread with my own code support- if the author chooses to implement and/or support it that is his business.

y2ksw

02-02-2011 06:34 PM

I have updated this product with a BETA 1.0.4 version (1.0.2 is stable), in order to allow testing the new features, such as multiple tags, reverse order, forum ID's, original file names and so stuff.

I have tested this on my production site with 800.000+ posts and it worked. However, until I have tested this on another, much larger board, I will not make it an official release.

Thanks to all who have made contributions and have tried it so far :)

vortodox

02-02-2011 06:42 PM

FYEAH! Thanks! I will test it :)

Xencored

02-02-2011 07:00 PM

OH my cant belive i missed this one Thanks muchy :)

vortodox

02-02-2011 07:10 PM

Installed, and it works fine for now :3
Thanks allot for this update!

Xencored

02-03-2011 12:57 PM

Hmm am not sure if its working or not lol. Does it not Cache external images than am confused :(

BirdOPrey5

02-03-2011 12:59 PM

Quote:

Originally Posted by Vodkaholic (Post 2158212)

Hmm am not sure if its working or not lol. Does it not Cache external images than am confused :(

Right click on images in your posts and check the URL to them... they should have local URLs (to your website) instead of the original URLs they had when you posted them.

It's not immediate, the task runs every 10 minutes and if you have a backlog of posts it could be days or weeks before you catch up to recent posts.

Xencored

02-03-2011 01:01 PM

Quote:

Originally Posted by BirdOPrey5 (Post 2158214)

So yea thats what i thought "like the old Cache mod he made"

Seems not to be working for me tho
Not sure if its because me forums is off tho

BirdOPrey5

02-03-2011 01:14 PM

Yeah it would probably only work with forums turned on because scheduled tasks won't run if off. You could manually run the scheduled task.

Xencored

02-03-2011 01:17 PM

Quote:

Originally Posted by BirdOPrey5 (Post 2158222)

Yeah it would probably only work with forums turned on because scheduled tasks won't run if off. You could manually run the scheduled task.

Yea i did manually a few times the other day still did'nt cache them tho hmm

y2ksw

02-04-2011 11:31 AM

Quote:

Originally Posted by Vodkaholic (Post 2158225)

Yea i did manually a few times the other day still did'nt cache them tho hmm

Please turn logging of that task on, and check the logs. If an image has been imported, you can check it clicking the link in the logs ... those images should be then stored locally.

After checking, disable the logging because it may flood your logs with a tremedous number of entries, depending on the number of pasted images ... ;)

And of course, tasks are not running until forums are on.

BirdOPrey5

02-04-2011 01:16 PM

Can you explain more about the "cache" option in the new Beta?

y2ksw

02-04-2011 06:13 PM

Quote:

Originally Posted by BirdOPrey5 (Post 2158688)

Can you explain more about the "cache" option in the new Beta?

Not much more than it says in the options. If the cache option is on (default), all image references are loaded into memory by a single query (and N fetches), if it is off, every image is individually searched in the database. Thus, if you have a large forum and a huge amount of images, PHP may run out of memory and the mod stops apparently to work ("could not allocate N bytes" error). In this case you have 2 options: empty the iei_img table, or switch caching off.

BirdOPrey5

02-04-2011 07:28 PM

Any estimate on what you'd consider a "huge amount" of images? 100,000? 1,000.000? 10,000,000? More than that?

y2ksw

02-06-2011 08:50 AM

Quote:

Originally Posted by BirdOPrey5 (Post 2158819)

Any estimate on what you'd consider a "huge amount" of images? 100,000? 1,000.000? 10,000,000? More than that?

That depends on the memory you are using for each PHP process and the length of the URL's. When the plugin stops working from one day to the other, you probably have reached this limit.

On one site we have 120.000+ images and it continues to work, thus maybe that's not huge. But we also give unlimited memory to the PHP processes and the server can hold up to 16 GB. Maybe a million is huge, in this context :)

BirdOPrey5

02-06-2011 12:10 PM

Thanks.

BadgerDog

02-06-2011 07:04 PM

1 Attachment(s)

Went live today with 1.04 .... :up:

Crossed my fingers and opened up most of our forums to start importing 10 posts at a time from the oldest to newest .... :D

First few runs seemed quicker with this ordering (old to new) than the testing we did with (new to old)... also turned OFF logging after first few runs ...

Let's see how long it takes to get through our site to the latest ...

y2ksw and BirdOPrey5 ... thanks for all the help from both of you ... :)

Regards,
Doug

Update: Ok, I started the live execution yesterday at about 4:00 pm (same time as this original post) and checked the first few logs. Everything seemed to be working according to the logs and the earliest threads seemed to be processed. Out of curiosity to see where it was at this morning, I turned LOGGING back ON and waited until cycle for cron had past. I checked logs and there wasn't anything there at all. So, I manually ran the process and watched it (see pic attached). It's been sitting like that for almost 15 minutes, just supposedly processing 10 posts, so I suspect it's hung and I have no idea how long it hasn't been running since I started it at 4:00 pm yesterday. I also don't know if there's been any database damage or not due to the hanging, but the vBulletin error logs don't show any SQL errors and the board seems to be running fine.

I assume I should quite the processing on the screen attached and do what next? :confused:

Update to update: In checking actually file storage on the server under /imported, it appears that the last pics imported happened at about 5:05 pm, so it appears that the import mod ran just over an hour after I had originally started it before it stopped working .... and hasn't worked since.

Another update: I have tried turning OFF the cache and that didn't make any difference.

Another update: For info purposes, we are running on a dedicated private host and Safe Mode is OFF, so that shouldn't be a time out problem. y2ksw suspects the job is timing out, so I've asked him for some debug code to be built into a test version of the cron job for manual run testing, so we can see where this is failing and perhaps why.

PhilG

02-11-2011 12:05 PM

I have just tested this and it's got a serious bug.

I have set "Forum URL replacement" to http://static.domain.com and alot of images get replaced with just that domain. For example there was an image "http://www.angelfire.com/oz/gbeze/images/circuit.gif" and it got replaced with "http://static.domain.com/

Apart from that great mod.

y2ksw

02-11-2011 12:25 PM

Quote:

Originally Posted by PhilG (Post 2161337)

Is your forum path starting with http://static.domain.com ? In other words, is your forum contained in the forum replacement or vice versa?

Could you please attach a setting with which I may reproduce this behaviour?

Thanks

sarangan

02-11-2011 06:51 PM

It would be better if there was an option to exclude forums instead of include forums. :-) Looking forward to have that option also in the feature version. A super and nice plugin. I love it.. Thanks a lot friend.. :-)

All times are GMT. The time now is 05:00 AM.

Page 4 of 22

« First

Last »

Show 40 post(s) from this thread on one page

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.01980 seconds Memory Usage 1,895KB Queries Executed 10 (?)
More Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (10)bbcode_php_printable (20)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)pagenav (1)pagenav_curpage (4)pagenav_pagelink (1)pagenav_pagelinkrel (1)post_thanks_navbar_search (1)printthread (40)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start pagenav_page pagenav_complete bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages: