vb.org Archive - Cache the datastore in the filesystem

Page 1 of 2

Show 40 post(s) from this thread on one page

vb.org Archive (https://vborg.vbsupport.ru/index.php)

- vBulletin 3.0 Beta Releases (https://vborg.vbsupport.ru/forumdisplay.php?f=34)

- - Cache the datastore in the filesystem (https://vborg.vbsupport.ru/showthread.php?t=75032)

Stadler

01-23-2005 10:00 PM

Cache the datastore in the filesystem

Ok, the purpose of this hack is to move the huge parts of the datastore,that are called each time you access a page, into a file unserialized and passed through var_export in order to save the time used to load the data from the database and to unserialize it. If you're using a PHP optimizer, like eAccelerator for example, this may save even more time I think. The result is, that huge arrays, like the forumcache are now loaded simply by executing

PHP Code:


		
			
require_once(DATASTORE_FILE_DIR.'datastore_cache.php');

in your init.php.

Additionally this saves 4 calls to the function unserialize, which is broken in PHP 5.0.3 and 4.3.10, so this should be a good work around until PHP 5.0.4 and/or PHP 4.3.11 has been released.

It tries to finish, what has been started in the code and AFAICS you can even find the beginnings of that in the init.php of vB3.0.6,although I'm still running vB 3.0.3, so I can't tell, that this works without problems in newer versions.

Some Notes:

I recommend to place rebuild_dscache.php into a protected directory and to alter the first chdir(...); in the script, so that no one can abuse this, by flooding your forum with calls to that script.
Additionally you should remove the backup-files named datastore_cache_TIMESTAMP.php in your includes dir (or wherever the datastore-cache is stored) from time to from to save disc-space.
Use this at your own risk and make sure to backup first before installing this hack.
This hack is not designed for a multi-webserver setup. So unless the data on the filesystem is being updated on the fly its not recommended to install this.

Changelog:
0.8.1 Alpha:

removed a relic of an old hack from dscache.txt
Added config_dscache.php
backups can be turned off now
you can change the directory, where the datastore_cache.php is stored now
Fixed a bug, that may cause unserialize-errors in the CP out of the blue
rebuild_dscache.php is verbosive now

0.8.2 Alpha:

Added CVS-Ids to some files
Added a note to rebuild_dscache.php
Added a note about PHP < 4.2.0 to dscache.txt

0.8.3 Alpha:

Removed some relics of the old backup functionality
some changes to make it possible to run it at the commandline (see below)
Allow to specify a key in the config-file, so this script will only run if you specify the correct key (-k KEY on the commandline or ?key=KEY in the URL)
You can make the script silent by adding -s respectively ?silent=1

1.0 (Final):

No changes at all. All files are the same as in Version 0.8.3 Alpha.

Erwin

01-24-2005 09:03 PM

Very very very cool. :) I must try this.

Andreas

01-24-2005 09:20 PM

Hehe, I did that a few days ago ;)
But I think there as not much difference in perfomace, once the bug in unserialize() has been fixed.
IIRC can can find discussions about this on vB.com, whre one of the Devs said that this is even slower then using the DB.

Erwin

01-24-2005 09:34 PM

That was what my concern was. But while php 4.3.10 has that serialize bug which does slow my site down, this is a nice workaround.

kall	01-24-2005 09:50 PM

From the guy who runs my host:

Quote:

Originally Posted by ASO

We've been fine and the actual places where unserialize are used in vB are relatively minimal compared to other areas of script execution. Believe me, it does not have as much of an impact as you may happen to think. And we'll upgrade as soon as a stable version is released (which I imagine would be soon).

Is he correct? Is it 'relatively minimal'?

(I tried this hack but it blew up in my face with foreach errors and stuff...I have apostrophes in my forum descriptions and it caused errors in functions_datastorecache.php ... )

Stadler

01-24-2005 10:31 PM

Quote:

Originally Posted by kall

From the guy who runs my host:

Is he correct? Is it 'relatively minimal'?

(I tried this hack but it blew up in my face with foreach errors and stuff...I have apostrophes in my forum descriptions and it caused errors in functions_datastorecache.php ... )

The impact is relative to the amount of forums and usergroups you have, since this could bloat up your forumcache. And I can't confirm your problems. We have forums with apostrpohes in the descriptions, too.

Quote:

IIRC can can find discussions about this on vB.com, whre one of the Devs said that this is even slower then using the DB.

mmh, then it's the connection between the webserver and the database, that slows down our forum, since I've recognized a difference of about 0.5s, after activating that hack. Our forum runs on a version of PHP, thats rather old, that is ...

nexialys

01-24-2005 10:39 PM

i'd like to see a performance ranking on this hack, hit to a db like the one here on vb.org ...

Acido

01-25-2005 01:16 AM

I cant found this code in includes/init.php

PHP Code:


		
			
 $optionsfound = !empty($vboptions);

Can you provide another location to add the hack code?

I'm using vbulletin 3.0.1

Thanx in advance.

Erwin

01-25-2005 09:53 AM

Mmm... you need to chmod 0777 the includes directory or you get a permission denied. Correct?

Erwin

01-25-2005 10:58 AM

Okay, got it working 100% with my other hacks. :) Thanks!!! This has helped my multi-server setup. Will wait till we are at peak times and compare...

nexialys

01-25-2005 12:42 PM

Quote:

Originally Posted by Acido

I cant found this code in includes/init.php

PHP Code:


		
			
 $optionsfound = !empty($vboptions);

search simply for $optionsfound ... if you don't have it, you have a strange vbulletin version, because this variable is there since 3.0... line 549 of the actual 3.0.6...

BTW, the instructions are strange.. maybe thefile is corrupted ?!
look:

Quote:

#-------------------==={FIND}===------------------#
################################################## #

$optionsfound = !empty($vboptions);

################################################## #
#-------------==={REPLACE IT WITH}===-------------#
################################################## #

// START Hack: datastore_cache
//- $optionsfound = !empty($vboptions);
if (is_array($vboptions) AND isset($vboptions['languageid']))
{
$optionsfound = true;
}
if ($url == 'index.php')
{
$url = "$vboptions[forumhome].php";
}
$versionnumber = &$vboptions['templateversion'];
// END Hack: datastore_cache
", $spiderpattern) . ')#';

################################################## #

what's that

Quote:

", $spiderpattern) . ')#';

Acido

01-25-2005 01:21 PM

Quote:

Originally Posted by nexialys

The File is OK. No problem there.

In my init.php i have 5 instances of $optionsfound (listing the more relevants):

PHP Code:


		
			
 // load options, special templates and default language
$optionsfound = false;
if (defined('VB3UPGRADE'))
.............

PHP Code:


		
			
// get $vboptions array
    case 'options':
    {
     $vboptions = unserialize($storeitem['data']);
     if (is_array($vboptions) AND isset($vboptions['languageid']))
     {
     $optionsfound = true;
     }
.............

PHP Code:


		
			
 // Fatal Error
if (!$optionsfound)
{
require_once('./includes/adminfunctions.php');
require_once('./includes/functions.php');
$vboptions = build_options(
..............

You think that i can try the hack in this init? Or i need wait to upgrade vbulletin?

nexialys

01-25-2005 01:47 PM

@Acido.. you have listed the less relevants... look at that one:
$optionsfound = !empty($vboptions);

you have to forget the tabs (just find the text, not the line)

Acido

01-25-2005 03:04 PM

Quote:

Originally Posted by nexialys

@Acido.. you have listed the less relevants... look at that one:
$optionsfound = !empty($vboptions);

you have to forget the tabs (just find the text, not the line)

My init.php have not that line. In fact, my board is running without the file datastore_cache.php

Not sure, but i think that your hack need 3.0.2 or higher to work. I'll try it after upgrade.

Thanx for your reply's.
:)

Guy G

01-25-2005 03:27 PM

Will this saqves queries?
If i dont have the unserialize bug will this effect?

Stadler

01-25-2005 04:01 PM

Quote:

Originally Posted by nexialys

BTW, the instructions are strange.. maybe thefile is corrupted ?!
look:

Quote:

", $spiderpattern) . ')#';

what's that

I took the file of an older hack and copy&pasted the changes for this hack into that file.

Its just a leftover of that hack and it will be removed in future versions.

Stadler

01-25-2005 04:07 PM

Quote:

Originally Posted by Guy G

Will this saqves queries?

No. Only the traffic between the Webserver and the database-server would be reduced a bit.

Quote:

If i dont have the unserialize bug will this effect?

In most cases: No.
If your database and your webserver are on different machines it may have an effect, depending on the connection between them.

Guy G

01-25-2005 04:08 PM

Quote:

Originally Posted by Stadler

No. Only the traffic between the Webserver and the database-server would be reduced a bit.In most cases: No.
If your database and your webserver are on different machines it may have an effect, depending on the connection between them.

Ok Thanks.

Stadler

01-25-2005 04:12 PM

Quote:

Originally Posted by Acido

I cant found this code in includes/init.php

PHP Code:


		
			
 $optionsfound = !empty($vboptions);

Can you provide another location to add the hack code?

I'm using vbulletin 3.0.1

Thanx in advance.

Please upgrade to vB 3.0.2 at least. You can try to use the init.php from a newer version and to copy datastore_cache.php.new to datastore_cache.php. But I'm not sure, if this will work.

Quote:

Originally Posted by Acido

See above :)

Stadler

01-25-2005 04:19 PM

Quote:

Originally Posted by Erwin

Mmm... you need to chmod 0777 the includes directory or you get a permission denied. Correct?

Yes. I'll add an option to change the directory, where the file is located in future versions.

Erwin

01-25-2005 05:33 PM

I look forward to future versions.

Interesting that the vB developers had already coded this into vB by default. You just needed to activate it in config.php really. :)

Stadler

01-25-2005 06:14 PM

Quote:

Originally Posted by Erwin

I look forward to future versions.

Interesting that the vB developers had already coded this into vB by default. You just needed to activate it in config.php really. :)

Not really. They just started adding this, but it won't work without any changes.

Stadler

01-25-2005 07:45 PM

OK, I'm finished with the new version so far:

0.8.1 Alpha:

removed a relic of an old hack from dscache.txt
Added config_dscache.php
backups can be turned off now
you can change the directory, where the datastore_cache.php is stored now
Fixed a bug, that may cause unserialize-errors in the CP out of the blue
rebuild_dscache.php is verbosive now

Be aware, that this hack is still an Alpha and needs testing.

Erwin

01-25-2005 08:16 PM

I installed 0.8.1 - works 100%. The backups were an annoyance and I'm glad you coded it so that they disappear. :) Thanks.

kall	01-26-2005 06:07 PM

Parse error: parse error, unexpected T_STRING, expecting ')' in /home3/nzboards/public_html/forums/includes/datastore_cache.php on line 2833

Line 2833: 'description' => 'Kall\\'s private forum',

Stadler

01-26-2005 07:57 PM

In your functions.php (the hacked version) find the following:

PHP Code:


		
			
            $data_code = preg_replace('/### (start|end) /', '\' . "\\x23\\x23\\x23 \1 " . \'', $data_code);

            $datastore_cache = file_get_contents(DATASTORE_FILE_DIR . 'datastore_cache.php');



            $datastore_cache = preg_replace(

                "/### start $title ###".

                "(?:[^#]*(?:#(?!## end $title ###)[^#]*)*)".

                "### end $title ###/",

                "### start $title ###\n\$$varname = $data_code;\n### end $title ###",

                $datastore_cache

            );

and replace it with the following:

PHP Code:


		
			
            $data_code = str_replace(

                array(

                    '### start ',

                    '### end ',

                    '{DATA_CODE}'

                ),

                array(

                    '\' . "\\x23\\x23\\x23 start " . \'',

                    '\' . "\\x23\\x23\\x23 end " . \'',

                    '\' . "\\x7bDATA_CODE\\x7d" . \''

                ), $data_code);



            $datastore_cache = file_get_contents(DATASTORE_FILE_DIR . 'datastore_cache.php');



            $datastore_cache = preg_replace(

                "/### start $title ###".

                "(?:[^#]*(?:#(?!## end $title ###)[^#]*)*)".

                "### end $title ###/",

                "### start $title ###\n\$$varname = {DATA_CODE};\n### end $title ###",

                $datastore_cache

            );

            $datastore_cache = str_replace('{DATA_CODE}', $data_code, $datastore_cache);

That should do the trick.

Andreas

01-26-2005 08:53 PM

Hmm ... why don't you just write the variables without comments?
I think this could make it easier :)

kall	01-26-2005 09:46 PM

Quote:

Originally Posted by Stadler

In your functions.php (the hacked version) find the following:

PHP Code:


		
			
            $data_code = preg_replace('/### (start|end) /', '\' . "\\x23\\x23\\x23 \1 " . \'', $data_code);

            $datastore_cache = file_get_contents(DATASTORE_FILE_DIR . 'datastore_cache.php');



            $datastore_cache = preg_replace(

                "/### start $title ###".

                "(?:[^#]*(?:#(?!## end $title ###)[^#]*)*)".

                "### end $title ###/",

                "### start $title ###\n\$$varname = $data_code;\n### end $title ###",

                $datastore_cache

            );

and replace it with the following:

PHP Code:


		
			
            $data_code = str_replace(

                array(

                    '### start ',

                    '### end ',

                    '{DATA_CODE}'

                ),

                array(

                    '\' . "\\x23\\x23\\x23 start " . \'',

                    '\' . "\\x23\\x23\\x23 end " . \'',

                    '\' . "\\x7bDATA_CODE\\x7d" . \''

                ), $data_code);



            $datastore_cache = file_get_contents(DATASTORE_FILE_DIR . 'datastore_cache.php');



            $datastore_cache = preg_replace(

                "/### start $title ###".

                "(?:[^#]*(?:#(?!## end $title ###)[^#]*)*)".

                "### end $title ###/",

                "### start $title ###\n\$$varname = {DATA_CODE};\n### end $title ###",

                $datastore_cache

            );

            $datastore_cache = str_replace('{DATA_CODE}', $data_code, $datastore_cache);

That should do the trick.

I think I will wait until the serialise bug is fixed...I have destroyed my forums twice now trying to implement this...it also doesn't seem to play nicely with Ushop.

Good idea though, and thanks for the help. :)

Erwin

01-27-2005 08:40 PM

This hack works but I've uninstalled it :). It was really annoying to have to keep manually updating the datastore on each of my separate web servers. The crunch came when I put my forums offline to do a backup, but as I did not update the datastore manually, members could still access the forums, causing my 15 Gb database to become corrupted. :) Took me 28 hours to fix it manually as I didn't want to lose even 1 day of data. There is a slight page loading improvement, but I'd rather have the convenience.

Stadler

01-27-2005 08:54 PM

Well yeah, if you're running this on a multi server setup, you need to synchronize everything on the fly and I suppose, that setting up such a system can be a lot of work, I guess.

Stadler

01-27-2005 09:08 PM

I've just added the following warning to the description:
This hack is not designed for a multi-server setup. So unless the data on the filesystem is being updated on the fly its not recommended to install this.

Thanks for helping with your feedback.

Erwin

01-27-2005 11:41 PM

Quote:

Originally Posted by Stadler

In fact, your hack is designed ESPECIALLY for separate web and database servers. :) It saves a lot of interserver traffic.

A good workaround is to make sure each time you edit settings in AdminCP the datastore is automatically rebuild. The problem is not multi-server, but multi-web server = when the datastore is rebuild, it only gets rebuild on the ONE web server. Trying to sync it in real-time won't be easy. :)

Stadler

02-03-2005 12:24 PM

Quote:

Originally Posted by KirbyDE

Hmm ... why don't you just write the variables without comments?
I think this could make it easier :)

I'm using these comments as delimiters right now and additionally these comments are in the file by default now. The delimiting comments ensure, that nothing is being replaced, if the regex itself fails, instead of messing up the whole file, causing more problems than it solves.

Stadler

02-03-2005 12:30 PM

Quote:

Originally Posted by Erwin

Yeah, you're right. I've changed multi-server to multi-webserver in the description. Thanks :)

weeno

02-03-2005 09:17 PM

anyone confirm if this works on 3.0.6 without obvious problems?

arn

Stadler

02-03-2005 10:12 PM

Quote:

Originally Posted by weeno

anyone confirm if this works on 3.0.6 without obvious problems?

arn

WFM

I've upgraded to vB 3.0.6 Wednesday evening and I had no problems so far.

btw: The backup feature may be removed in future version. I usually implemented it, so you could restore a previous version, if something went wrong, but since rebuild_dscache.php works perfectly, these backups are no longer needed.

Erwin

02-05-2005 03:32 AM

Quote:

Originally Posted by weeno

anyone confirm if this works on 3.0.6 without obvious problems?

arn

Yes, I got it working on 3.0.6 too. :) But I don't currently use it.

kall	02-13-2005 05:53 PM

Is anyone running this on a forum that has Ushop?

Erwin

02-13-2005 08:56 PM

Quote:

Originally Posted by kall

Is anyone running this on a forum that has Ushop?

I got it working with my version of ushop - you basically need to add the ushop variables to datastore templates too.

mark99

02-16-2005 12:53 PM

Ok, I have a few questions before I decided whether to risk sticking it on my production forum. It installed fine on my beta test and appeared to cut the page loading time down from 25/30seconds to 15.

My server has been hit hard by vB3's agressive database load and the unserialize() bug, hence the obvious difference. However, some aspects confuse me, for example:

Quote:

This hack works but I've uninstalled it . It was really annoying to have to keep manually updating the datastore on each of my separate web servers. The crunch came when I put my forums offline to do a backup, but as I did not update the datastore manually, members could still access the forums, causing my 15 Gb database to become corrupted.

My site (e.g: blahblah.co.uk) calls a database (e.g: db001.hosting.co.uk) thats given its own domain name by the same web host, but I'm worried about the above quote.

1) When do you actually need to run the 'rebuild' process? Judging by the data - only when you change an option in the admin CP or add/edit one of your forums/category names & options. Does it need to be rebuilt after every new post or is it just doing the more static data?

2) Can I remove the admincp/rebuild....php script from the server when not needing to be used or does something require it to be there?

3) If I need to take my forums offline to change the options, then (as per above) will this corrupt my database? I'm confused about how this can be prevented as admins always need to update options/add forums etc. now and then. Do you take it offline, run the rebuild, update the options, run the rebuild, put it online and then run the rebuild again? Am a little confused ;) .

-----------

Otherwise, can I just say, this is an excellent idea. If your database gets a lot of load and your actual webserver is still fast, then calling the key settings/forum names+details from the file system instead of the database (I think that's how it works?) saves a lot of DB pressure.

This may be more obvious to those on shared hosting, when vB3's heavy load can become a pain.

All times are GMT. The time now is 04:43 PM.

Page 1 of 2

Show 40 post(s) from this thread on one page

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.01593 seconds Memory Usage 1,901KB Queries Executed 10 (?)
More Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (11)bbcode_php_printable (25)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)pagenav (1)pagenav_curpage (1)pagenav_pagelink (1)post_thanks_navbar_search (1)printthread (40)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start pagenav_page pagenav_complete bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages: