Cache the datastore in the filesystem

[Stadler]

Ok, the purpose of this hack is to move the huge parts of the datastore,that are called each time you access a page, into a file unserialized and passed through var_export in order to save the time used to load the data from the database and to unserialize it. If you're using a PHP optimizer, like eAccelerator for example, this may save even more time I think. The result is, that huge arrays, like the forumcache are now loaded simply by executingrequire_once(DATASTORE_FILE_DIR.'datastor e_cache.php');in your init.php.

Additionally this saves 4 calls to the function unserialize, which is broken in PHP 5.0.3 and 4.3.10, so this should be a good work around until PHP 5.0.4 and/or PHP 4.3.11 has been released.

It tries to finish, what has been started in the code and AFAICS you can even find the beginnings of that in the init.php of vB3.0.6,although I'm still running vB 3.0.3, so I can't tell, that this works without problems in newer versions.

Some Notes:
I recommend to place rebuild_dscache.php into a protected directory and to alter the first chdir(...); in the script, so that no one can abuse this, by flooding your forum with calls to that script.
Additionally you should remove the backup-files named datastore_cache_TIMESTAMP.php in your includes dir (or wherever the datastore-cache is stored) from time to from to save disc-space.
Use this at your own risk and make sure to backup first before installing this hack.
This hack is not designed for a multi-webserver setup. So unless the data on the filesystem is being updated on the fly its not recommended to install this.
Changelog:
0.8.1 Alpha:

removed a relic of an old hack from dscache.txt
Added config_dscache.php
backups can be turned off now
you can change the directory, where the datastore_cache.php is stored now
Fixed a bug, that may cause unserialize-errors in the CP out of the blue
rebuild_dscache.php is verbosive now

0.8.2 Alpha:

Added CVS-Ids to some files
Added a note to rebuild_dscache.php
Added a note about PHP < 4.2.0 to dscache.txt

0.8.3 Alpha:

Removed some relics of the old backup functionality
some changes to make it possible to run it at the commandline (see below)
Allow to specify a key in the config-file, so this script will only run if you specify the correct key (-k KEY on the commandline or ?key=KEY in the URL)
You can make the script silent by adding -s respectively ?silent=1

1.0 (Final):

No changes at all. All files are the same as in Version 0.8.3 Alpha.

[Erwin]

Very very very cool. :) I must try this.

[Andreas]

Hehe, I did that a few days ago ;)
But I think there as not much difference in perfomace, once the bug in unserialize() has been fixed.
IIRC can can find discussions about this on vB.com, whre one of the Devs said that this is even slower then using the DB.

[Erwin]

That was what my concern was. But while php 4.3.10 has that serialize bug which does slow my site down, this is a nice workaround.

[kall]

From the guy who runs my host:

We've been fine and the actual places where unserialize are used in vB are relatively minimal compared to other areas of script execution. Believe me, it does not have as much of an impact as you may happen to think. And we'll upgrade as soon as a stable version is released (which I imagine would be soon).
Is he correct? Is it 'relatively minimal'?

(I tried this hack but it blew up in my face with foreach errors and stuff...I have apostrophes in my forum descriptions and it caused errors in functions_datastorecache.php ... )

[Stadler]

From the guy who runs my host:

Is he correct? Is it 'relatively minimal'?

(I tried this hack but it blew up in my face with foreach errors and stuff...I have apostrophes in my forum descriptions and it caused errors in functions_datastorecache.php ... )
The impact is relative to the amount of forums and usergroups you have, since this could bloat up your forumcache. And I can't confirm your problems. We have forums with apostrpohes in the descriptions, too.IIRC can can find discussions about this on vB.com, whre one of the Devs said that this is even slower then using the DB.mmh, then it's the connection between the webserver and the database, that slows down our forum, since I've recognized a difference of about 0.5s, after activating that hack. Our forum runs on a version of PHP, thats rather old, that is ...

[nexialys]

i'd like to see a performance ranking on this hack, hit to a db like the one here on vb.org ...

[Acido]

I cant found this code in includes/init.php

$optionsfound = !empty($vboptions);

Can you provide another location to add the hack code?

I'm using vbulletin 3.0.1

Thanx in advance.

[Erwin]

Mmm... you need to chmod 0777 the includes directory or you get a permission denied. Correct?

[Erwin]

Okay, got it working 100% with my other hacks. :) Thanks!!! This has helped my multi-server setup. Will wait till we are at peak times and compare...

[nexialys]

I cant found this code in includes/init.php $optionsfound = !empty($vboptions); search simply for $optionsfound ... if you don't have it, you have a strange vbulletin version, because this variable is there since 3.0... line 549 of the actual 3.0.6...

BTW, the instructions are strange.. maybe thefile is corrupted ?!
look:#-------------------==={FIND}===------------------#
################################################## #

$optionsfound = !empty($vboptions);

################################################## #
#-------------==={REPLACE IT WITH}===-------------#
################################################## #

// START Hack: datastore_cache
//- $optionsfound = !empty($vboptions);
if (is_array($vboptions) AND isset($vboptions['languageid']))
{
$optionsfound = true;
}
if ($url == 'index.php')
{
$url = "$vboptions[forumhome].php";
}
$versionnumber = &$vboptions['templateversion'];
// END Hack: datastore_cache
", $spiderpattern) . ')#';

################################################## #

what's that ", $spiderpattern) . ')#';

[Acido]

search simply for $optionsfound ... if you don't have it, you have a strange vbulletin version, because this variable is there since 3.0... line 549 of the actual 3.0.6...

BTW, the instructions are strange.. maybe thefile is corrupted ?!
look:

what's thatThe File is OK. No problem there.

In my init.php i have 5 instances of $optionsfound (listing the more relevants):

// load options, special templates and default language
$optionsfound = false;
if (defined('VB3UPGRADE'))
.............


// get $vboptions array
case 'options':
{
$vboptions = unserialize($storeitem['data']);
if (is_array($vboptions) AND isset($vboptions['languageid']))
{
$optionsfound = true;
}
.............

// Fatal Error
if (!$optionsfound)
{
require_once('./includes/adminfunctions.php');
require_once('./includes/functions.php');
$vboptions = build_options(
..............


You think that i can try the hack in this init? Or i need wait to upgrade vbulletin?

[nexialys]

@Acido.. you have listed the less relevants... look at that one:
$optionsfound = !empty($vboptions);

you have to forget the tabs (just find the text, not the line)

[Acido]

@Acido.. you have listed the less relevants... look at that one:
$optionsfound = !empty($vboptions);

you have to forget the tabs (just find the text, not the line)
My init.php have not that line. In fact, my board is running without the file datastore_cache.php

Not sure, but i think that your hack need 3.0.2 or higher to work. I'll try it after upgrade.

Thanx for your reply's.
:)

[Guy G]

Will this saqves queries?
If i dont have the unserialize bug will this effect?

[Stadler]

BTW, the instructions are strange.. maybe thefile is corrupted ?!
look:
", $spiderpattern) . ')#'; what's thatI took the file of an older hack and copy&pasted the changes for this hack into that file.

Its just a leftover of that hack and it will be removed in future versions.

[Stadler]

Will this saqves queries?No. Only the traffic between the Webserver and the database-server would be reduced a bit.If i dont have the unserialize bug will this effect?In most cases: No.
If your database and your webserver are on different machines it may have an effect, depending on the connection between them.

[Guy G]

No. Only the traffic between the Webserver and the database-server would be reduced a bit.In most cases: No.
If your database and your webserver are on different machines it may have an effect, depending on the connection between them.
Ok Thanks.

[Stadler]

I cant found this code in includes/init.php

$optionsfound = !empty($vboptions);

Can you provide another location to add the hack code?

I'm using vbulletin 3.0.1

Thanx in advance.Please upgrade to vB 3.0.2 at least. You can try to use the init.php from a newer version and to copy datastore_cache.php.new to datastore_cache.php. But I'm not sure, if this will work.My init.php have not that line. In fact, my board is running without the file datastore_cache.php

Not sure, but i think that your hack need 3.0.2 or higher to work. I'll try it after upgrade.

Thanx for your reply's.
:)See above :)

[Stadler]

Mmm... you need to chmod 0777 the includes directory or you get a permission denied. Correct?Yes. I'll add an option to change the directory, where the file is located in future versions.

[Erwin]

I look forward to future versions.

Interesting that the vB developers had already coded this into vB by default. You just needed to activate it in config.php really. :)

[Stadler]

I look forward to future versions.

Interesting that the vB developers had already coded this into vB by default. You just needed to activate it in config.php really. :)
Not really. They just started adding this, but it won't work without any changes.

[Stadler]

OK, I'm finished with the new version so far:

0.8.1 Alpha:

removed a relic of an old hack from dscache.txt
Added config_dscache.php
backups can be turned off now
you can change the directory, where the datastore_cache.php is stored now
Fixed a bug, that may cause unserialize-errors in the CP out of the blue
rebuild_dscache.php is verbosive now
Be aware, that this hack is still an Alpha and needs testing.

[Erwin]

I installed 0.8.1 - works 100%. The backups were an annoyance and I'm glad you coded it so that they disappear. :) Thanks.

[kall]

Parse error: parse error, unexpected T_STRING, expecting ')' in /home3/nzboards/public_html/forums/includes/datastore_cache.php on line 2833

Line 2833: 'description' => 'Kall\\'s private forum',

[Stadler]

In your functions.php (the hacked version) find the following: $data_code = preg_replace('/### (start|end) /', '\' . "\\x23\\x23\\x23 \1 " . \'', $data_code);
$datastore_cache = file_get_contents(DATASTORE_FILE_DIR . 'datastore_cache.php');

$datastore_cache = preg_replace(
"/### start $title ###".
"(?:[^#]*(?:#(?!## end $title ###)[^#]*)*)".
"### end $title ###/",
"### start $title ###\n\$$varname = $data_code;\n### end $title ###",
$datastore_cache
);and replace it with the following: $data_code = str_replace(
array(
'### start ',
'### end ',
'{DATA_CODE}'
),
array(
'\' . "\\x23\\x23\\x23 start " . \'',
'\' . "\\x23\\x23\\x23 end " . \'',
'\' . "\\x7bDATA_CODE\\x7d" . \''
), $data_code);

$datastore_cache = file_get_contents(DATASTORE_FILE_DIR . 'datastore_cache.php');

$datastore_cache = preg_replace(
"/### start $title ###".
"(?:[^#]*(?:#(?!## end $title ###)[^#]*)*)".
"### end $title ###/",
"### start $title ###\n\$$varname = {DATA_CODE};\n### end $title ###",
$datastore_cache
);
$datastore_cache = str_replace('{DATA_CODE}', $data_code, $datastore_cache);
That should do the trick.

[Andreas]

Hmm ... why don't you just write the variables without comments?
I think this could make it easier :)

[kall]

In your functions.php (the hacked version) find the following: $data_code = preg_replace('/### (start|end) /', '\' . "\\x23\\x23\\x23 \1 " . \'', $data_code);
$datastore_cache = file_get_contents(DATASTORE_FILE_DIR . 'datastore_cache.php');

$datastore_cache = preg_replace(
"/### start $title ###".
"(?:[^#]*(?:#(?!## end $title ###)[^#]*)*)".
"### end $title ###/",
"### start $title ###\n\$$varname = $data_code;\n### end $title ###",
$datastore_cache
);and replace it with the following: $data_code = str_replace(
array(
'### start ',
'### end ',
'{DATA_CODE}'
),
array(
'\' . "\\x23\\x23\\x23 start " . \'',
'\' . "\\x23\\x23\\x23 end " . \'',
'\' . "\\x7bDATA_CODE\\x7d" . \''
), $data_code);

$datastore_cache = file_get_contents(DATASTORE_FILE_DIR . 'datastore_cache.php');

$datastore_cache = preg_replace(
"/### start $title ###".
"(?:[^#]*(?:#(?!## end $title ###)[^#]*)*)".
"### end $title ###/",
"### start $title ###\n\$$varname = {DATA_CODE};\n### end $title ###",
$datastore_cache
);
$datastore_cache = str_replace('{DATA_CODE}', $data_code, $datastore_cache);
That should do the trick.
I think I will wait until the serialise bug is fixed...I have destroyed my forums twice now trying to implement this...it also doesn't seem to play nicely with Ushop.

Good idea though, and thanks for the help. :)

[Erwin]

This hack works but I've uninstalled it :). It was really annoying to have to keep manually updating the datastore on each of my separate web servers. The crunch came when I put my forums offline to do a backup, but as I did not update the datastore manually, members could still access the forums, causing my 15 Gb database to become corrupted. :) Took me 28 hours to fix it manually as I didn't want to lose even 1 day of data. There is a slight page loading improvement, but I'd rather have the convenience.

[Stadler]

Well yeah, if you're running this on a multi server setup, you need to synchronize everything on the fly and I suppose, that setting up such a system can be a lot of work, I guess.

[Stadler]

I've just added the following warning to the description:
This hack is not designed for a multi-server setup. So unless the data on the filesystem is being updated on the fly its not recommended to install this.

Thanks for helping with your feedback.

[Erwin]

I've just added the following warning to the description:
This hack is not designed for a multi-server setup. So unless the data on the filesystem is being updated on the fly its not recommended to install this.

Thanks for helping with your feedback.
In fact, your hack is designed ESPECIALLY for separate web and database servers. :) It saves a lot of interserver traffic.

A good workaround is to make sure each time you edit settings in AdminCP the datastore is automatically rebuild. The problem is not multi-server, but multi-web server = when the datastore is rebuild, it only gets rebuild on the ONE web server. Trying to sync it in real-time won't be easy. :)

[Stadler]

Hmm ... why don't you just write the variables without comments?
I think this could make it easier :)I'm using these comments as delimiters right now and additionally these comments are in the file by default now. The delimiting comments ensure, that nothing is being replaced, if the regex itself fails, instead of messing up the whole file, causing more problems than it solves.

[Stadler]

In fact, your hack is designed ESPECIALLY for separate web and database servers. :) It saves a lot of interserver traffic.

A good workaround is to make sure each time you edit settings in AdminCP the datastore is automatically rebuild. The problem is not multi-server, but multi-web server = when the datastore is rebuild, it only gets rebuild on the ONE web server. Trying to sync it in real-time won't be easy. :)Yeah, you're right. I've changed multi-server to multi-webserver in the description. Thanks :)

[weeno]

anyone confirm if this works on 3.0.6 without obvious problems?

arn

[Stadler]

anyone confirm if this works on 3.0.6 without obvious problems?

arn
WFM

I've upgraded to vB 3.0.6 Wednesday evening and I had no problems so far.

btw: The backup feature may be removed in future version. I usually implemented it, so you could restore a previous version, if something went wrong, but since rebuild_dscache.php works perfectly, these backups are no longer needed.

[Erwin]

anyone confirm if this works on 3.0.6 without obvious problems?

arn
Yes, I got it working on 3.0.6 too. :) But I don't currently use it.

[kall]

Is anyone running this on a forum that has Ushop?

[Erwin]

Is anyone running this on a forum that has Ushop?
I got it working with my version of ushop - you basically need to add the ushop variables to datastore templates too.

[mark99]

Ok, I have a few questions before I decided whether to risk sticking it on my production forum. It installed fine on my beta test and appeared to cut the page loading time down from 25/30seconds to 15.

My server has been hit hard by vB3's agressive database load and the unserialize() bug, hence the obvious difference. However, some aspects confuse me, for example:

This hack works but I've uninstalled it . It was really annoying to have to keep manually updating the datastore on each of my separate web servers. The crunch came when I put my forums offline to do a backup, but as I did not update the datastore manually, members could still access the forums, causing my 15 Gb database to become corrupted.

My site (e.g: blahblah.co.uk) calls a database (e.g: db001.hosting.co.uk) thats given its own domain name by the same web host, but I'm worried about the above quote.

1) When do you actually need to run the 'rebuild' process? Judging by the data - only when you change an option in the admin CP or add/edit one of your forums/category names & options. Does it need to be rebuilt after every new post or is it just doing the more static data?

2) Can I remove the admincp/rebuild....php script from the server when not needing to be used or does something require it to be there?

3) If I need to take my forums offline to change the options, then (as per above) will this corrupt my database? I'm confused about how this can be prevented as admins always need to update options/add forums etc. now and then. Do you take it offline, run the rebuild, update the options, run the rebuild, put it online and then run the rebuild again? Am a little confused ;) .

-----------

Otherwise, can I just say, this is an excellent idea. If your database gets a lot of load and your actual webserver is still fast, then calling the key settings/forum names+details from the file system instead of the database (I think that's how it works?) saves a lot of DB pressure.

This may be more obvious to those on shared hosting, when vB3's heavy load can become a pain.

[mark99]

If somebody could help with the above post then I'd be most grateful, it's a good solution for easing DB load.

[kall]

I got it working with my version of ushop - you basically need to add the ushop variables to datastore templates too.
Could you please share how you managed it? :)

[Stadler]

1) When do you actually need to run the 'rebuild' process? Judging by the data - only when you change an option in the admin CP or add/edit one of your forums/category names & options. Does it need to be rebuilt after every new post or is it just doing the more static data?

2) Can I remove the admincp/rebuild....php script from the server when not needing to be used or does something require it to be there?

3) If I need to take my forums offline to change the options, then (as per above) will this corrupt my database? I'm confused about how this can be prevented as admins always need to update options/add forums etc. now and then. Do you take it offline, run the rebuild, update the options, run the rebuild, put it online and then run the rebuild again? Am a little confused ;)Do you have a single or multiple webservers, where your forum is running on? If your forum only runs on a single werserver, theres nothing to worry about.

PS: Sorry for the late reply. I've been busy with other projects recently.

[Stadler]

I have updated some files. I've only updated some of the documentation and no functionality so far, so you don't need to update anything. You'll only waste your time ;)

For people with PHP < 4.2.0 I've added a note, what they could do to get the hack running.

Erwin: Scott pointed me to the following: How about triggering rebuild_dscache.php on all your server with something, like
fopen('http://www01.example.com/path/to/rebuild_dscache.php?silent=1&key=SECRET');
?

PS: PHP 4.3.11RC1 and PHP 5.0.4RC1 are out: http://news.php.net/php.general/210866

[Erwin]

Could you please share how you managed it? :)
editing init.php and adding the ushop datastore into the cache too. :)

[nexialys]

SUGG: drop the config_dstore.php file and add the $dscache_dir var in the config.php define('CACHE_DATASTORE_FILE', true);
define('DATASTORE_FILE_DIR', './includes/');

$dscache_dobackup and BACKUP_DATASTORE_FILE are useless for now... would need to set it somewhere or delete it!

also, for security, isn't it suggested to NOT chmod the includes dir ?!

[Stadler]

$dscache_dobackup and BACKUP_DATASTORE_FILE are useless for now... would need to set it somewhere or delete it!D'Oh, right. I removed the backups from the script but forgot to mention that, sry. I'll remove the redundant code.

However: I'm working on enabling rebuild_dscache.php for running as a shell script. You will then be able to specify a key, so the script will only run, if you call it with rebuild_dscache.php?k=sdiztgejklt for example.
Or with php rebuild_dscache.php -k sdiztgejklt on your favorite shell.

Another parameter would be ?silent=1 or -s to make the script silent, except error-messages.

[Stadler]

Ok, can anyone test the recent version?
It works fine on my test-system (commandline and through URI)

Changes:
0.8.3 Alpha:

Removed some relics of the old backup functionality
some changes to make it possible to run it at the commandline (see below)
Allow to specify a key in the config-file, so this script will only run if you specify the correct key (-k KEY on the commandline or ?key=KEY in the URL)
You can make the script silent by adding -s respectively ?silent=1

[Acido]

OK, now i have this hack installed.
But i cant see any datastore_cache_TIMESTAMP.php file in my includes dir (i'm usign defaults dirs of the hack).

How i can to know if this is working?

[Stadler]

OK, now i have this hack installed.
But i cant see any datastore_cache_TIMESTAMP.php file in my includes dir (i'm usign defaults dirs of the hack).

How i can to know if this is working?
These are relics of the old backup functionality and won't be created anymore. I've left the note about them for users who still have old backup files in their includes-dir (or whereever they are storing their datastore_cache.php.

You should take a look at datastore_cache.php. It should contain the expanded arrays of the usergroupcache or the forumcache for example.

[The Realist]

When I run the rebuild_dscache.php script I get this error:

Starting to rebuild the datastore cache ...

Copying datastore_cache.php.new to datastore_cache.php ...
Warning: copy(./includes/datastore_cache.php): failed to open stream: Permission denied in /home/xxxxxxxxxxxx/public_html/forums/admincp/rebuild_dscache.php on line 96

failed

Line 96 says: if (!copy(DATASTORE_FILE_DIR . 'datastore_cache.php.new', DATASTORE_FILE_DIR . 'datastore_cache.php'))

Any ideas?

[nexialys]

you simply forgot to chmod the file and add the datastore_cache.php.new to your directory... (so you did not followed the instruction completely.. ;) )

[The Realist]

I did chmod the file and added all the files as per instructions.

:ogre:

you simply forgot to chmod the file and add the datastore_cache.php.new to your directory... (so you did not followed the instruction completely.. ;) )

[Stadler]

I did chmod the file and added all the files as per instructions.

:ogre:
Maybe your webserver hasn't the permission to overwrite datastore_cache.php? If that file doesn't exist in your includes-dir, then you need to chmod the directory to 755, 775 or 777 depending on your setup (777, if you're in doubt). If datastore_cache.php exists, it should be enough to chmod it to 644, 664 or 666 depending on your setup Assuming that file doesn't need to be executable).

[Trigunflame]

This is prob one of the most "relevant" useful modifcations on vbulletin.org right now, can't believe I didn't get around to installing it until today.

Hope vbulletin fully implements this at a later date as stock code.

[kall]

editing init.php and adding the ushop datastore into the cache too. :)
If you could maybe hold my hand and lead me through the process step by step please?

(I haven't the foggiest idea what you mean to do..)

[weeno]

Anyone have thoughts on the ongoing use of this in light of PHP 4.3.11 released.

I've upgraded to 4.3.11 but am keeping the hack installed... as it must still reduce the burden on the server to cache this.

arn

[Stadler]

I'm using it because the connection between the database and the webserver is a bottleneck.

People should see for their own, if this reduces network traffic and/or speeds up connections significantly or not. If it doesn't, its recommended to uninstall or disable this hack.

[Stadler]

Since I had no issues for a long time and since no further issues were reported I've bumped the Version number to 1.0 now. All files are unchanged from version 0.8.3 Alpha, so there no need to upgrade.

Could someone move this to the proper forum please?

Regards,
Christian Stadler

[nowrap]

danke stadler!
is working like a charme on our vB 3.0.7 board having a 4MB forumcache!
rendertimes dropped from 1.2s+ to 0.3s and keeping the load at bay.

running:
apache 2.0.54
php 4.4.1
eaccelerator 0.9.4

[bogene2020]

Stadler,
Look at this https://vborg.vbsupport.ru/showthread.php?t=100486

Ok I was trying to figure out how many forums/sub forums I can have before I start running into performance problems. from what I have come to understand after I get to about 1500 Forums I will start to experience problems because of all the stuff that loads . Will it be safe to say that a hack like this datastore hack will help improve performance when one has so many forums . Thanks

[Stadler]

Stadler,
Look at this https://vborg.vbsupport.ru/showthread.php?t=100486

Ok I was trying to figure out how many forums/sub forums I can have before I start running into performance problems. from what I have come to understand after I get to about 1500 Forums I will start to experience problems because of all the stuff that loads . Will it be safe to say that a hack like this datastore hack will help improve performance when one has so many forums . Thanks

If your webserver and your database are on the same server, that hack would slow down your forum because of the time needed to load, parse and compile the script in that file. Maybe it is faster in combination with a PHP-Accelerator, like eAccelerator (which is recommended on such a big forum as yours), but that should be marginally. In short: The hack is mostly useless, if webserver and database are on the same server.

If your webserver and your database are on different servers, this hack would reduce your network-traffic between both machines, thus speeding up your forum (in most cases).

However: I can't guarantee anything, because there are so many different setups. You have to see for yourself, if this hack helps you or not. Or you could possibly try vB 3.5, where this feature is built in and much more improved, than this hack (For example you can choose to store the data directly in the memory).

Regards,
Christian Stadler