Cache System Explanation (datastore)
 

I'm going to try and explain the datastore system used in vBulltin. This tutorial is based around users who are already familar with PHP and MySQL.

What is Cache?
A memory area where frequently accessed data can be stored for rapid access. (View Definitions). Basicly, you can store information into the database, and grab all this information with just using 1 query. Some may question this and say: "Isn't all information in the database retrived by queires?". Yup, all information in the database is retrived using MySQL queries, though the method used is diffrent when using the datastore system in vBulletin.

I'll show you way using the datastore is a better solution for you and the users who will use your hacks.

Datastore Method
Lets say you want to use a drop down form with 3 options to choose from. Obviously you would need to create alittle script to add, delete, and edit the information; and when you make this script. Where does the info go? Well it would go into the database. Lets call the MySQL table "dropdownoptions" with 3 fields. ID, Name, and Value.

Attachment 44423

Now everytime you add, edit, or delete your information, it will be changed in the database. Here is an example of the database tree. We have 3 rows of information.

• ID = (1) Name = (Dog) Value(Dog)
• ID = (2) Name = (Cat) Value(Cat)
• ID = (3) Name = (Cow) Value(Cow)

Attachment 44424

After you insert this information into the database, the next step would be to insert this info in the datastore. You can do that like this.

Let me go ahead and tell you exacly what the above does. What we did was just grab all in the information in the database from the table dropdownmenu. The while() function will loop the information from the begging to the end. While it's getting all this information, It's storing it all into the variable named $variable_array.

Then you need to use the function called build_datastore(). This function requires the following. build_datastore(1, 2). #1 is the name of your datastore item, and number two is the serialize information to store. This function will create a new datastore item with the name and all your info.

The serialize info will look like this...
So now we have all our infomation stored in another area of the database, in one row. Heres a picture so you can compair it to the other ones.

Attachment 44425

Don't let the serialize data freak you out. You don't really NEED to read that data. Though I kinda like doing it, so I'll break it down alittle for you. Here is a tree of the serialize data.

In PHP it would look like this.
So now we have the info stored, lets get it back out in a readable formate. :p This is the easy part. This is the code I use.

This will print the following:
(ID: 1) (Title: Dog) (Value: Dog)
(ID: 2) (Title: Cat) (Value: Cat)
(ID: 3) (Title: Cow) (Value: Cow)

What your doing is simple. The variable $vbulletin->dropmenu holds the information though it's still serialize. The reason why $vbulletin->dropmenu is the variable is because you need to tell vBulletin which datastore row to get. In this case it is dropmenu because remember we stored it in there like this: build_datastore('dropmenu', serialize($variable_array));

Now in order for vBulletin to know about that specific row, you need to add it to the $specialtemplates array. Example:

So now that you understand where the variable is coming from, we need to unserialize the data, cause it looks all ugly and weird.

So $vbulletin->dropmenu = unserialize($vbulletin->dropmenu); baiscly gets the serialize info and unserialize's it using the function called unserialize() Once that is finished you want to loop the info using a foreach() function. foreach ($vbulletin->dropmenu AS $dropmenu). Doing that is storing the array info into $dropmenu and you can get each one by using the following vaiables. $dropmenu['id'], $dropmenu['title'], and $dropmenu['value'].

The reason why this is better is because vBulletin allready runs one global query to get all the datatore information. So instead of running a query everytime you want to get the drop down information, you just get it from the datastore and save that query. Some may think this is kinda extreme when you can just run a query, though as your site grows; you want to save as many queires as possiable.

If your interested in saving queries and bandwidth. I would suggest taking a look at Trigunflames profile. He has released many hacks to help in these areas. :)

Excellent article!

The only thing I have to say, "Damn, these thumbnail size!"

great, now I can understand how it can be used :)

Great article. Thank you, LiveWire.

very, very, great tutorial 'ken' ;)

huh now i get what the a:3 means its the size of the array. damn i feel dumb now.

Nice. Is it useful write my own items to file-based datastore cache?
If yes, how I can do this ?

right..

I'm being thick here, clearly :

I want to show $numbermembers and $totalposts at the top of each page of my forum, from within the header template.

As these two values are not available unless you've already parsed the forumhome stats, I've successfully cached these values just after they are calculated at the end of index.php.

I have added it to special templates on the index.php page.

The problem I'm having is getting the data back out again, and into a useable format so I can insert it into the template.

I'm missing a step somewhere..

If I stick $vbulletin->cachedstats into the header template I see the full serialised data in there, but I can't work out from the examples exactly how (and probably more importantly WHERE) to unserialise the data and turn it into eval variables that I can use in the header template.

If anyone could give me a bit of code and tell me where to put it, I would be eternally greatful!

Thanks in advance.

Simon

You need $vbulletin->cachedstats = unserialize($vbulletin->cachedstats);
for all data (except loaded from file based datastore in my case).
store full parsed html code for maximum speed.
If not, use like this in templates :
Total users : {$vbulletin->cachedstats[totalusers]}

Thank you netwind.

Could you please tell me where I can put the unserialise code?

I've tried lots of places, but wherever I'm placing it seems to be either before the cache memory is populated, or after the header has been built.

cheers

Read beginning from words
Now in order for vBulletin to know about that specific row, you need to add it to the $specialtemplates array.
Try parse_templates hook.

LOL

I'd already added it to $specialtemplates, as stated in my original post :)

Thank you very much for your help, the parse_templates hook worked a treat.

The problem I have now, is having to manually add 'cachestats' to the special templates array of every single page. Is there a way to maybe add to this array from within global.php or something?

Thanks again

Simon

aah..

answered my own question (wonders will never cease!)

I've added :
$specialtemplates[] = 'cachestats';

to the top of the global.php file which works. So will play around to find the right hook to put it in.

--

additional :
It seems that there isn't a hook early enough in global.php
I'm really playing around in the dark here, so if anyone knows of a hook which runs early enough, could you please let me know?

Cheers
Simon

Hook global_start for $specialtempates[] .=
and parse_templates (and later called specific hooks) for unserialize and other jobs

LOL

what a muppet.. thanks mate :)

correction..

global_start appears to be way too late. There are two that seem to run earlier than this, and neither of those are early enough either.

ok.. I've whacked the $specialtemplates[] = ... into global_start..

but the variable is empty when the header template is built.

when I had identical set-up, but instead of global_start, I hacked the line into the start of the global.php file it worked fine.

Sorry, this my tip was for custom stand-alone scripts like this :
I think modifying global.php only one way..

ok.

Thanks a lot for your help netwind, it got me working anyway, with only one file edit.

If anyone else knows a hook I can use, it would be great.

Thanks

Simon

I have another related problem: build html in admincp/ not work,
since style not selected. I need example code which build all special vars for function fetch_template() in default style.

I must say this is a GREAT article, but I need a little help building the forumcache. If I do as this article says, it just f*cks up my forum and its unable to load correctly :(

I believe forumcache has its own function.

Like build_options(), build_datastore(), build_birthdays(), build_maxloggedin(),
perhaps build_forumcache() ?

I think this thread only applies to datastore objects you have created yourself.

I can't find any similar functions that work for the forumcache. Actually build_datastore() works fine saving the data, but I think the input is wrong :(

This was a great article and it explained the concept very well. I do have a minor quibble in your example. Rather than name= dog, value=dog, it might help people understand better if the var name and var value were different, e.g. name=dog, value=terrier.

Something I'd add to the discussion is that cron and serialize go hand in hand. The best place to use serialize is when accessing data frequently, therefore using vb's cached datastore w/o adding a query...but the updating of data is relatively infrequent. A classic way to update that data periodically is cron.
https://vborg.vbsupport.ru/showthrea...highlight=cron

I really hope they add a hook that will allow us to add to $specialtemplates before it's actually used. It's kind of a bummer to have to hack up a file over something so simple.

Not to suggest that adding the hook is simple. I think there's some initialization for the hook system in the datastore, or something along those lines. It's just a pain and it would be cool if they stumble across some fix in the future.

I think the trick around that is with the "array_merge()" function. It's been awhile so I'm not completely sure, but I believe it's due to the $specialtemplates array being called before global.php.

It's because the query that uses $specialtemplates and populates the datastore variables comes before global_start. I don't really understand why, but I think it has something to do with the hook system not being able to work until the datastore info is available.

Have you tried the init_startup hook? There is another Datastore fetch executed after it I think. This isn't tested but you could maybe try:

Hook Location: init_startup

Awesome, thanks man. I haven't tested it either, but it looks like that should work.

--------------- Added [DATE]1194736320[/DATE] at [TIME]1194736320[/TIME] ---------------

Ok, it did work. It added the extra query, but it works.

You have to be careful and make sure it's like this though:

Doing it without both sets of quotes makes everything (including the admincp) inaccessbile until you go in and correct it in both the plugin table and the datastore entry. They try to warn against it in the code, but I wasn't exactly sure what it meant.

Ooops yup you need to put it in quotes because otherwise MySQL will read it as a column name instead of a value.

I'll correct it in my original post.

The Function build_datastore also has a 3rd parameter which should be mentioned, as it seems all serialized data in the datastore table requires a value of 1...

By setting the 3rd param to true, it's no longer necessary to run unserialize like this:
$vbulletin->dropmenu = unserialize($vbulletin->dropmenu);

There'fore we can just access the info from the datastore like this:

<font color="darkgreen">What if I have my own class object and I want to build cache acessible from it only and not use the $vbulletin object?

Like ...
$myclass->mycache['something']

instead of $vbulletin->mycache['something']

How would I change it so its like that?</font>

With a couple tweaks (probably because this is 2 years old...), i've successfully got my first datastore plugin working :)

Thanks for the write up Ken. Never experienced with the datastore much.

Can you please share it with us. I am so tired of trying to make this work. It has been 5 days but I couldn't cache anything yet. It would be great to know how to cache sql queries.

i'm still trying to figure out how to export this entire thing (with all my crons/ phrases etc) but no one is helping me out on .com
http://www.vbulletin.com/forum/showthread.php?t=274552
otherwise, i'd just post my entire product.


basically, this is what i did.

1) I made a cron job to populate a table from live post/thread table data. (Perhaps this table is useless now altogether, but i like having a table i can reference during testing. next version probably won't include this cache_ table at all)

Again, you can skip truncating and building the table... just run the big query here as the $query = part to get your array/data.


Set up the cron job to run this script as you like.
Run it once manually to make sure you have data in the datastore row.

there will actually be a row with the name skm_active_cache in the datastore table.


2) from there, i have 4 plugins in my product
a) cache_templates hook : contains the simple caching of my custom template which prints 1 row of data (ie, <tr><td>1</td><td>2</td></tr> ) similar to the 'threadbits' template

it contains a simple line:
b) init_startup hook : special templates is probably a mis-nomer at this point, as i'm NOT using the special templates array merge style.
This file pulls the datastore row into memory:
note the quotes are important. This name should match what you ran in your cron under the build_datastore() function.

c) forumhome_complete hook : parse the data and print it to the template step. obviously, forumhome_complete should match the area you are trying to hook into. in my case, it was on the forum home.

d) the template will look for the $output array to print, ala:
e) the template group is not required... just makes things easier to group when you have a lot of templates.


That's about it.

basically, you need to fill the datastore with your query, and pull with a foreach of the array once the data has been initialized.

Hello,

Thank you so much for your tutorial. This was exactly what I was looking for. I write my own products/plugins (our forums is huge) and I was wondering how to cache stuff around.

However I faced one problem: I couldn't read data back.

The solution was to add a "1" parameter on the store command:

build_datastore('name',serialize($variable),1);

When retrieving you don't need to "unserialize" it.

Also I was trying to call my cached array inside a class that hasn't had $vbulletin as a global variable, so I needed to add:

global $vbulletin;

On the top of my plugin.

And I also needed to add a plugin for the init_startup hook, adding:

$datastore_fetch[]="'name'";

I hope this feedback helps other coders.

Cheers,
Gabriel.

This should really be added to the first post. Took forever to figure this out. :)

Is there a datastore per user?

If you have this, now How can I delete one of the data (red below) ???

it could be possible to fetch more than 1 value

for example

(ID: 1) (Title: Dog) (Value: Dog , dog2, dog4)

and be able to delete dog4

(ID: 1) (Title: Dog) (Value: Dog , dog2)

or add more

any ideas?

maybe first check if its empty or :S?

Hello,

I seem to be unable to persist the values in datastore. Variables built and stored to datastore in global_start plugin are lost in subsequent requests.

I wish the purpose, usage, lifecycle, limitations and performance of datastore would be discussed more in the article.

-Pete

Thank you for this, was pulling my hair out my head trying to get this to work.
That's what i was missing :)