Remote image caching script v1 (and questions)

[brunod76]

Hiya!

Do you have a lot of images on your forum which come from external sites? Then read this!

I had an idea recently. My forum needs to display many images from other websites. Basicly it's a stock investing forum, and we display many chart images from big websites such as Yahoo Finance, StockCharts, etc.

The images are dynamic (cannot just put them as attachment) because these charts change every day, but I also don't like linking to their images directly as this is not very bandwith friendly for these sites. So comes in my idea: caching locally these remote images.

So I've created a very simple replacement, wherever there is an IMG bb code, it looks at the URL to check if it's local or remote (on my domain or not), if not it loads the image, saves it locally in a cache folder, and from there on displays it from the cache folder. There is a configurable cache delay expiry that will reload the image from the source after a certain delay of time (for example every 30 minutes reload direcly from the source).

The other really nice feature is that there is absolutely no change for your users. The urls to remote images they have entered in IMG tags are replaced with a call to the caching script directly.

Example:

With an image tag such as :
[img]/url/to/your/image[/img]

You get:
[img]cache.php?url=/url/to/your/image[/img]

In cache.php, it verifies if the image is already cached and if the cache delay is not expired, if not it loads the image and saves it to the local cache folder, and if it is it simply loads it.

Pros:
- If you plan to display many images from remote servers this is a bandwith friendly way of displaying these images.
- Bypasses hotlinking protections. The images are loaded by your server and then served from your server, so hotlinking protections will not complain (since this is just like if a normal user is viewing the image directly from the origin website.
- Consistancy: everything loads from YOUR server, so you won't have images taking longer than others to load (since different hosts have different speeds).
- If ever the remote image gets erased, you'll still have it in your cache so it will still display.
- URLs stay the same as the original ones, so if the image gets changed remotely it will refresh itself on your server too, so this is a better alternative than 'saving it' locally.

Cons:
- On first load (if image is not found in cache or cache delay expired) it will take a small bit longer than normally, but once it's cached it usually faster
- Puts a bit more load because you have a script executing for each IMG tag being loaded. Bare in mind this only applies for remote images only (NOT those stored on your own server)

I have finished the first version (which works with version 3.5.1 of vBulletin by the way) but there should be some improvements. Mostly I was waiting for some input from the experienced coders in here on how to be integrate this code. I'm not familiar yet with the structure of the code in vBulletin, so I made a separate .php file and edited a couple lines directly in the class that handles bb codes. I realize that this is probably not the best way, so input would be appreciated.

So here's my code. Create a file called cache.php containing:

<?php
// Remote caching script v1.0

if(!defined('_CACHE_DIR_URL')) {
	define('_CACHE_DIR_URL', '/url/to/cache/');  // The url to your cache dir
	define('_CACHE_DIR_PATH', 'C:\\path_to\\your\\htdocs\\cache\\');	// The physical path to your cache dir
	define('_CACHE_DURATION_SECONDS', 300);		// Number of seconds needed to expire files
	define('_DEBUG', false);	// Set debugging display on / off. If true, it will display text instead of loading the image to explain what's happening.'
}


if(isset($_GET['url'])) {
	$url = $_GET['url'];
	
	if(isset($url) && $url != "") {
		if(isExternalUrl($url)) {
			// this is an external url, get it from local cache
			$newurl = @getFromLocalCache($url);
			debug("New url: $newurl");
			if(isset($newurl)) {
				cacheSuccess("307 Temporary Redirect", $newurl);
			} else {
				cacheFailure("500 Internal Server Error", $url);
			}
		
		} else {
			// This is a local file, so simply load it normally (no cache)
			cacheSuccess("301 Moved Permanently", $url);
		}
	
	} else {
		// The url argument is not set.
		cacheFailure("400 Bad Request", 'null');
	}
		
} else {
	// The url argument is not set.
	cacheFailure("400 Bad Request", 'null');
}


//////////// FUNCTIONS DEFINITIONS ///////////////

function debug($msg) {
	if(_DEBUG) {
		echo "$msg<br>";
	}
}

function cacheSuccess($code, $url) {
	if(_DEBUG) {
		debug("cache success - headers= code: $code, url: $url");
	} else {
		header("HTTP/1.1 $code");
		header("Location: $url");
	}
}

function cacheFailure($err, $url) {
	if(_DEBUG) {
		debug("cache failure - $err, url: $url");
	} else {
		// force the start url
		cacheSuccess('307 Temporary Redirect', $url);
	}
}

// Checks if the url is located on another host than the local one.
function isExternalUrl($url) {
	$url_stuff = parse_url($url);
	return isset($url_stuff['host']) && $url_stuff['host'] != $_SERVER['HTTP_HOST'];
}

function isCached($filename) {
	$isCached = false;
	$filename = _CACHE_DIR_PATH . $filename;
	
	if(file_exists($filename)) {
		// Check to see if file has been modified in last $cachelength seconds
		$fileinfo = stat($filename);
		if (time() - _CACHE_DURATION_SECONDS < $fileinfo["mtime"]) {
			// file is less than 5 mintes old - return file pointer to the image
			$isCached = true;
		}
	}

	return $isCached;
}

function getEncodedUrl($url) {
	return str_replace(array('/', '&', '?', ':', '-', '='), '', $url);
}

function getFromLocalCache($source) {
	$localFileName = getEncodedUrl($source);
	debug("local file: $localFileName");	
	
	if(!isCached($localFileName)) {
		debug("NOT cached, downloading...");
		$success = downloadUrlToFile($source, _CACHE_DIR_PATH . $localFileName);
		if($success) {
			return _CACHE_DIR_URL . $localFileName;
		}
	} else {
		debug("file was in cache already");	
		return _CACHE_DIR_URL . $localFileName;
	}
	
	return;
	
}

function downloadUrlToFile($url, $destFile) {
	debug("downloading file $url -> $destFile");
		
	$read = fopen ( $url , "r" ); 
	if ( !$read ) { 
		debug("Could not open $url");
		return false;
	} 
	
	$HTML_output = ""; 
	
	// read the file
	while ($line = fgets ( $read , 256 )) { 
		$HTML_output.= $line; 
//		echo $line; 
	} 
	fclose ($read); 
	
	// finally, save the file's output 
	// in a cache file. 
	$write = fopen ($destFile , "w"); 
	
	if (!$write) { 
		// you might not have permission 
		// to write in that directory. 
		debug("could not open $destFile for writing");
		return false;
	} 
	
	// lock the write file and 
	// write all the contents into it 
	if ( !flock ( $write , LOCK_EX + LOCK_NB )) { 
		// for PHP version < 4.0.1 
		// change LOCK_EX to 2 
		debug("could not lock $destFile"); 
		return false;
	} 
	
	fwrite ( $write , $HTML_output , strlen ( $HTML_output ) ); 
	flock ( $write , LOCK_UN ); 
	
	// release lock. For PHP version < 4.0.1 
	// change LOCK_UN to 3 
	fclose ( $write ); 
	
	return true;
}

  	
?>

Edit the config variables up top to suit your server. You have to give appropriate write access to the cache folder of course since the server will store image files in there. Then, to try it out, load it such as

http://yourserver.com/cache.php?url=...mages/logo.gif

I suggest you put debug mode on first, and you should get a display such as:

local file: httpwww.google.caintlen_caimageslogo.gif
NOT cached, downloading...
downloading file http://www.google.ca/intl/en_ca/images/logo.gif -> C:\Program Files\xampp\htdocs\cache\httpwww.google.caintlen_caimageslogo.gif
New url: /mos/cache/httpwww.google.caintlen_caimageslogo.gif
cache success - headers= code: 307 Temporary Redirect, url: /mos/cache/httpwww.google.caintlen_caimageslogo.gif

Then hit refresh, the second time you should see:

local file: httpwww.google.caintlen_caimageslogo.gif
file was in cache already
New url: /mos/cache/httpwww.google.caintlen_caimageslogo.gif
cache success - headers= code: 307 Temporary Redirect, url: /mos/cache/httpwww.google.caintlen_caimageslogo.gif

Confirming that your image file was stored in the local cache. Please note that the image file name is simply the url with the special characters (/ etc) removed (to have a compliant file name).

If you have an error, you will see it displayed in debug mode. Most likely the errors will be with permissions to write files / wrong paths.

Once you got it working, turn debug off and you will see the image displayed.

To use it in a normal html image tag, you would do:

<img src="http://yourserver.com/cache.php?url=http://www.google.ca/intl/en_ca/images/logo.gif" />

Now to enable that into your vBulletin, this is the code you need in the file /includes/class_bbcode.php. Somewhere in this file you have the function handle_bbcode_img_match:

///////////////// CUSTOMIZED ///////////////////////////////////////
	function bd_isExternalUrl($url) {
		$url_stuff = parse_url($url);
		return isset($url_stuff['host']) && $url_stuff['host'] != $_SERVER['HTTP_HOST'];
	}
//////////////////////////////////////////////////////////////////////////

	/**
	* Handles a match of the [img] tag that will be displayed as an actual image.
	*
	* @param	string	The URL to the image.
	*
	* @return	string	HTML representation of the tag.
	*/
	function handle_bbcode_img_match($link)
	{
		$link = $this->strip_smilies(str_replace('\\"', '"', $link));

		// remove double spaces -- fixes issues with wordwrap
		$link = str_replace('  ', '', $link);

///////////////// CUSTOMIZED ///////////////////////////////////////
		if($this->bd_isExternalUrl($link)) {
			return '<img src="cache.php?url=' .  $link . '" border="0" alt="" />';
		} else {
			return '<img src="' .  $link . '" border="0" alt="" />';
		}
//////////////////////////////////////////////////////////////////////////
	}

Just insert the function bd_isExternalUrl just before like shown here which will recognize if the image is loaded exteranlly or not.

Thoughts / comments would be appreciated I hope you guys find this useful, I know I will!!

---------------

Update: one quick note, I know of a bug already and would like to know how to fix this. Basicly, the replacement works correctly, but when you edit the post and add a new image (where there was an image already) the previous image tags are broken.

I think it replaces the brackets ( [ and ] ) with their html equivalent or something. Anyways, you will not see the image anymore but rather the text of the bb code. I'm sure this is a very small problem and a more experience vB coder could help me fix this

[Turbosport]

Very nice m8

You need to find the reversemarkup.