Cache the datastore in the filesystem

Ok, the purpose of this hack is to move the huge parts of the datastore,that are called each time you access a page, into a file unserialized and passed through var_export in order to save the time used to load the data from the database and to unserialize it. If you're using a PHP optimizer, like eAccelerator for example, this may save even more time I think. The result is, that huge arrays, like the forumcache are now loaded simply by executing

PHP Code:

require_once(DATASTORE_FILE_DIR.'datastore_cache.php'); 


in your init.php.

Additionally this saves 4 calls to the function unserialize, which is broken in PHP 5.0.3 and 4.3.10, so this should be a good work around until PHP 5.0.4 and/or PHP 4.3.11 has been released.

It tries to finish, what has been started in the code and AFAICS you can even find the beginnings of that in the init.php of vB3.0.6,although I'm still running vB 3.0.3, so I can't tell, that this works without problems in newer versions.

Some Notes:

• I recommend to place rebuild_dscache.php into a protected directory and to alter the first chdir(...); in the script, so that no one can abuse this, by flooding your forum with calls to that script.
• Additionally you should remove the backup-files named datastore_cache_TIMESTAMP.php in your includes dir (or wherever the datastore-cache is stored) from time to from to save disc-space.
• Use this at your own risk and make sure to backup first before installing this hack.
• This hack is not designed for a multi-webserver setup. So unless the data on the filesystem is being updated on the fly its not recommended to install this.

Changelog:
0.8.1 Alpha:

• removed a relic of an old hack from dscache.txt
• Added config_dscache.php
• backups can be turned off now
• you can change the directory, where the datastore_cache.php is stored now
• Fixed a bug, that may cause unserialize-errors in the CP out of the blue
• rebuild_dscache.php is verbosive now

0.8.2 Alpha:

• Added CVS-Ids to some files
• Added a note to rebuild_dscache.php
• Added a note about PHP < 4.2.0 to dscache.txt

0.8.3 Alpha:

• Removed some relics of the old backup functionality
• some changes to make it possible to run it at the commandline (see below)
• Allow to specify a key in the config-file, so this script will only run if you specify the correct key (-k KEY on the commandline or ?key=KEY in the URL)
• You can make the script silent by adding -s respectively ?silent=1

1.0 (Final):

• No changes at all. All files are the same as in Version 0.8.3 Alpha.

[Stadler]

I've just added the following warning to the description:
This hack is not designed for a multi-server setup. So unless the data on the filesystem is being updated on the fly its not recommended to install this.

Thanks for helping with your feedback.

[Erwin]

Originally Posted by Stadler

I've just added the following warning to the description:
This hack is not designed for a multi-server setup. So unless the data on the filesystem is being updated on the fly its not recommended to install this.

Thanks for helping with your feedback.

In fact, your hack is designed ESPECIALLY for separate web and database servers. It saves a lot of interserver traffic.

A good workaround is to make sure each time you edit settings in AdminCP the datastore is automatically rebuild. The problem is not multi-server, but multi-web server = when the datastore is rebuild, it only gets rebuild on the ONE web server. Trying to sync it in real-time won't be easy.

[Stadler]

Originally Posted by KirbyDE

Hmm ... why don't you just write the variables without comments?
I think this could make it easier

I'm using these comments as delimiters right now and additionally these comments are in the file by default now. The delimiting comments ensure, that nothing is being replaced, if the regex itself fails, instead of messing up the whole file, causing more problems than it solves.

[Stadler]

Originally Posted by Erwin

In fact, your hack is designed ESPECIALLY for separate web and database servers. It saves a lot of interserver traffic.

A good workaround is to make sure each time you edit settings in AdminCP the datastore is automatically rebuild. The problem is not multi-server, but multi-web server = when the datastore is rebuild, it only gets rebuild on the ONE web server. Trying to sync it in real-time won't be easy.

Yeah, you're right. I've changed multi-server to multi-webserver in the description. Thanks

[weeno]

anyone confirm if this works on 3.0.6 without obvious problems?

arn

[Stadler]

Originally Posted by weeno

anyone confirm if this works on 3.0.6 without obvious problems?

arn

WFM

I've upgraded to vB 3.0.6 Wednesday evening and I had no problems so far.

btw: The backup feature may be removed in future version. I usually implemented it, so you could restore a previous version, if something went wrong, but since rebuild_dscache.php works perfectly, these backups are no longer needed.

[Erwin]

Originally Posted by weeno

anyone confirm if this works on 3.0.6 without obvious problems?

arn

Yes, I got it working on 3.0.6 too. But I don't currently use it.

[kall]

Is anyone running this on a forum that has Ushop?

[Erwin]

Originally Posted by kall

Is anyone running this on a forum that has Ushop?

I got it working with my version of ushop - you basically need to add the ushop variables to datastore templates too.

[mark99]

Ok, I have a few questions before I decided whether to risk sticking it on my production forum. It installed fine on my beta test and appeared to cut the page loading time down from 25/30seconds to 15.

My server has been hit hard by vB3's agressive database load and the unserialize() bug, hence the obvious difference. However, some aspects confuse me, for example:

This hack works but I've uninstalled it . It was really annoying to have to keep manually updating the datastore on each of my separate web servers. The crunch came when I put my forums offline to do a backup, but as I did not update the datastore manually, members could still access the forums, causing my 15 Gb database to become corrupted.

My site (e.g: blahblah.co.uk) calls a database (e.g: db001.hosting.co.uk) thats given its own domain name by the same web host, but I'm worried about the above quote.

1) When do you actually need to run the 'rebuild' process? Judging by the data - only when you change an option in the admin CP or add/edit one of your forums/category names & options. Does it need to be rebuilt after every new post or is it just doing the more static data?

2) Can I remove the admincp/rebuild....php script from the server when not needing to be used or does something require it to be there?

3) If I need to take my forums offline to change the options, then (as per above) will this corrupt my database? I'm confused about how this can be prevented as admins always need to update options/add forums etc. now and then. Do you take it offline, run the rebuild, update the options, run the rebuild, put it online and then run the rebuild again? Am a little confused .

-----------

Otherwise, can I just say, this is an excellent idea. If your database gets a lot of load and your actual webserver is still fast, then calling the key settings/forum names+details from the file system instead of the database (I think that's how it works?) saves a lot of DB pressure.

This may be more obvious to those on shared hosting, when vB3's heavy load can become a pain.