Cache the datastore in the filesystem

Ok, the purpose of this hack is to move the huge parts of the datastore,that are called each time you access a page, into a file unserialized and passed through var_export in order to save the time used to load the data from the database and to unserialize it. If you're using a PHP optimizer, like eAccelerator for example, this may save even more time I think. The result is, that huge arrays, like the forumcache are now loaded simply by executing

PHP Code:

require_once(DATASTORE_FILE_DIR.'datastore_cache.php'); 


in your init.php.

Additionally this saves 4 calls to the function unserialize, which is broken in PHP 5.0.3 and 4.3.10, so this should be a good work around until PHP 5.0.4 and/or PHP 4.3.11 has been released.

It tries to finish, what has been started in the code and AFAICS you can even find the beginnings of that in the init.php of vB3.0.6,although I'm still running vB 3.0.3, so I can't tell, that this works without problems in newer versions.

Some Notes:

• I recommend to place rebuild_dscache.php into a protected directory and to alter the first chdir(...); in the script, so that no one can abuse this, by flooding your forum with calls to that script.
• Additionally you should remove the backup-files named datastore_cache_TIMESTAMP.php in your includes dir (or wherever the datastore-cache is stored) from time to from to save disc-space.
• Use this at your own risk and make sure to backup first before installing this hack.
• This hack is not designed for a multi-webserver setup. So unless the data on the filesystem is being updated on the fly its not recommended to install this.

Changelog:
0.8.1 Alpha:

• removed a relic of an old hack from dscache.txt
• Added config_dscache.php
• backups can be turned off now
• you can change the directory, where the datastore_cache.php is stored now
• Fixed a bug, that may cause unserialize-errors in the CP out of the blue
• rebuild_dscache.php is verbosive now

0.8.2 Alpha:

• Added CVS-Ids to some files
• Added a note to rebuild_dscache.php
• Added a note about PHP < 4.2.0 to dscache.txt

0.8.3 Alpha:

• Removed some relics of the old backup functionality
• some changes to make it possible to run it at the commandline (see below)
• Allow to specify a key in the config-file, so this script will only run if you specify the correct key (-k KEY on the commandline or ?key=KEY in the URL)
• You can make the script silent by adding -s respectively ?silent=1

1.0 (Final):

• No changes at all. All files are the same as in Version 0.8.3 Alpha.

[Erwin]

I look forward to future versions.

Interesting that the vB developers had already coded this into vB by default. You just needed to activate it in config.php really.

[Stadler]

Originally Posted by Erwin

I look forward to future versions.

Interesting that the vB developers had already coded this into vB by default. You just needed to activate it in config.php really.

Not really. They just started adding this, but it won't work without any changes.

[Stadler]

OK, I'm finished with the new version so far:

0.8.1 Alpha:

• removed a relic of an old hack from dscache.txt
• Added config_dscache.php
• backups can be turned off now
• you can change the directory, where the datastore_cache.php is stored now
• Fixed a bug, that may cause unserialize-errors in the CP out of the blue
• rebuild_dscache.php is verbosive now

Be aware, that this hack is still an Alpha and needs testing.

[Erwin]

I installed 0.8.1 - works 100%. The backups were an annoyance and I'm glad you coded it so that they disappear. Thanks.

[kall]

Parse error: parse error, unexpected T_STRING, expecting ')' in /home3/nzboards/public_html/forums/includes/datastore_cache.php on line 2833

Line 2833: 'description' => 'Kall\\'s private forum',

[Stadler]

In your functions.php (the hacked version) find the following:

PHP Code:

            $data_code = preg_replace('/### (start|end) /', '\' . "\\x23\\x23\\x23 \1 " . \'', $data_code);
            $datastore_cache = file_get_contents(DATASTORE_FILE_DIR . 'datastore_cache.php');

            $datastore_cache = preg_replace(
                "/### start $title ###".
                "(?:[^#]*(?:#(?!## end $title ###)[^#]*)*)".
                "### end $title ###/",
                "### start $title ###\n\$$varname = $data_code;\n### end $title ###",
                $datastore_cache
            ); 


and replace it with the following:

PHP Code:

            $data_code = str_replace(
                array(
                    '### start ',
                    '### end ',
                    '{DATA_CODE}'
                ),
                array(
                    '\' . "\\x23\\x23\\x23 start " . \'',
                    '\' . "\\x23\\x23\\x23 end " . \'',
                    '\' . "\\x7bDATA_CODE\\x7d" . \''
                ), $data_code);

            $datastore_cache = file_get_contents(DATASTORE_FILE_DIR . 'datastore_cache.php');

            $datastore_cache = preg_replace(
                "/### start $title ###".
                "(?:[^#]*(?:#(?!## end $title ###)[^#]*)*)".
                "### end $title ###/",
                "### start $title ###\n\$$varname = {DATA_CODE};\n### end $title ###",
                $datastore_cache
            );
            $datastore_cache = str_replace('{DATA_CODE}', $data_code, $datastore_cache); 


That should do the trick.

[Andreas]

Hmm ... why don't you just write the variables without comments?
I think this could make it easier

[kall]

Originally Posted by Stadler

In your functions.php (the hacked version) find the following:

PHP Code:

            $data_code = preg_replace('/### (start|end) /', '\' . "\\x23\\x23\\x23 \1 " . \'', $data_code);
            $datastore_cache = file_get_contents(DATASTORE_FILE_DIR . 'datastore_cache.php');

            $datastore_cache = preg_replace(
                "/### start $title ###".
                "(?:[^#]*(?:#(?!## end $title ###)[^#]*)*)".
                "### end $title ###/",
                "### start $title ###\n\$$varname = $data_code;\n### end $title ###",
                $datastore_cache
            ); 


and replace it with the following:

PHP Code:

            $data_code = str_replace(
                array(
                    '### start ',
                    '### end ',
                    '{DATA_CODE}'
                ),
                array(
                    '\' . "\\x23\\x23\\x23 start " . \'',
                    '\' . "\\x23\\x23\\x23 end " . \'',
                    '\' . "\\x7bDATA_CODE\\x7d" . \''
                ), $data_code);

            $datastore_cache = file_get_contents(DATASTORE_FILE_DIR . 'datastore_cache.php');

            $datastore_cache = preg_replace(
                "/### start $title ###".
                "(?:[^#]*(?:#(?!## end $title ###)[^#]*)*)".
                "### end $title ###/",
                "### start $title ###\n\$$varname = {DATA_CODE};\n### end $title ###",
                $datastore_cache
            );
            $datastore_cache = str_replace('{DATA_CODE}', $data_code, $datastore_cache); 


That should do the trick.

I think I will wait until the serialise bug is fixed...I have destroyed my forums twice now trying to implement this...it also doesn't seem to play nicely with Ushop.

Good idea though, and thanks for the help.

[Erwin]

This hack works but I've uninstalled it . It was really annoying to have to keep manually updating the datastore on each of my separate web servers. The crunch came when I put my forums offline to do a backup, but as I did not update the datastore manually, members could still access the forums, causing my 15 Gb database to become corrupted. Took me 28 hours to fix it manually as I didn't want to lose even 1 day of data. There is a slight page loading improvement, but I'd rather have the convenience.

[Stadler]

Well yeah, if you're running this on a multi server setup, you need to synchronize everything on the fly and I suppose, that setting up such a system can be a lot of work, I guess.