Look at the top of just about every vbulletin script. You'll see something like this:
That pulls the datastore items into the code you are going to use. All you have to do after that is refer to them.

@Marco (and of course also all others interested)
I've re-implemented both Suggestions to cut down File Edits.

Only 1 File Edit (2 Lines) to config.php is required, and it will even survive Upgrades.

What do you think?

For Alternative 2:
Maintaining the Table should be done by Product Install/Uninstall-Codes so we don't have to care for that, except disabling/enabling a Product (which is handled by a Plugin).

I think i'll also implement Revans Idea (var_dump).

So far I've had to add two datastore items and one phrasegroup via edit to global.php On my site that doesn't bother me, but it will discourage me from sharing which is a shame :(

So you definitely have my full support trying to work this one out, but to be honest my knowledge is more patched together from hacking rather than being up to speed to follow the intricacies of trying to optimise how to do this.

Basically: sorry I ain't much practical help but folk are interested & keen to have a good solution to this (Y)

Thanks for the lesson, it worked like a charm :)
I even managed to code a workaround for having to prefix my custom datastore items with 'rpg_' (for ease of uninstall, identification etc) to stop my custom class calls to have to have this prefix as well :)

Putting them in global should only be done if they are used everywhere. If they are only used in a few scripts, they should be placed in those scripts, meaning maybe a lot of edits.

Will have a look at your solutions on monday kirby.

Monday has almost passed ... at least in my Timezone ;)

Yeah i know, and i didn't hae time today. :(
And i even doubt i will have time tomorrow, but will look as fast as i can.

But this don't need to stop other coders to have a look and comment. :D

*bump*

?

I really think the solution to this problem could be very simple

Datastore class:

[sql]
$dataitems = $db->query_read("
SELECT title, data
FROM " . TABLE_PREFIX . "datastore
# LEFT JOIN " . TABLE_PREFIX . "_NEWTABLE_ ON(datastore.title = _NEWTABLE_.title AND \"" . THIS_SCRIPT . "\" IN(_NEWTABLE_.page)
# LEFT JOIN " . TABLE_PREFIX . "product ON (product.productid = _NEWTABLE_.productid)
WHERE title IN ($itemlist)
# OR ( NOT NULL(_NEWTABLE_.title) AND product.active = true )
");
[/sql]

That's basically my idea :)

But it does only work for the default Datastore Class - and only for Datastore Items.

But since templates can already be cached via a hook, why does this system need to deal with that?

Similar modifications could be made to other datastore types, though they would probably have to drop the active checking, although, if you set a product as inactive, it could write to the other datastore methods at that point.

The end user doesnt need a script to add rows to the new table, the product xml file should deal with that bit automatically.

Phrasegroups for Example currently can't be done with Hooks (at least not for Guests).

I don't really get how you would go about implementing this Idea with other Datastore Classes.
Maybe some example Code?

Looking at the other classes, it shouldnt make any difference.

The other datastore types (file based or memcached) appear to fetch a datastore item if it doesnt exist in their specific storage mode and add it. Meaning it only needs to be uncached for one page load, then its added.

Im not sure how it deals with updated datastore items. I guess the build_datastore() functions deal with that too.

Didnt think of phrasegroups, but it shouldnt be hard to add phrasegroups either - Take a look at the deletion log table and how it is joined into multiple different types of items that could be deleted (posts, threads, pms)

Add a new column to the new table of type

[sql]
AND _NEWTABLE_.type = 'datastore' / 'phrasegroup'
[/sql]

Right. But how would the Classes determine if an item has to be cached if it's not in $specialtemplates?
That's the important question :)

Hope it doesnt come across as arrogant - but it doesnt matter. (From what I see from something like the Memcached class, all datastore items that have ever been "queried" are added to the Memcached cache, but a single query happens if it doesnt exist, but only the first time).

From what I see, the $specialtemplates variable is only useful if you want to save queries to the database while using the traditional datastore method.

If the item doesnt exist in the memcached/filestore storage, it will be queried. But only once. After it has been queried, it will be added to the relevant storage.

The biggest issue I see is that what happens when the datastore item is updated in the database. I do not see how it will update the item in the other storage methods - I suspect that it gets updated when you call the build_* functions. I havent looked at that area because I use the database method :)

Seems like you don't understand me, or I don't understand you :)

Let's say a Hack needs attachmentcache which isn't in $specialtemplates for the Script.
What should the Plugin do in order to get this?

Maybe just a crossed wire :)

1) The plugin that accesses a hook should do the standard thing it does to access the datastore item. (IIRC, $vbulletin->name) - however, it may need to be unserialized. I dont see an issue with unserializing in global_start.

2) The product should add a row into the NEWTABLE table with relevant details of the datastore item. Name, Product ID, Scripts that the datastore item should be loaded on (THIS_SCRIPT names) and any other details that are needed - havent fully thought of them.

At this point, the datastore class for DB type will query the datastore table joining the new table into it looking for conditions that match to load extra items (or if it exists in $specialtemplates). See query I posted above.

For the other datastore types, (because there are no queries involved for other types) it should be as simple as querying the datastore table once (an extra query for one page load) to load the datastore item into the other types' storage method.

There is an issue with this in that if the datastore item updates in the datastore, the product will have to deal with it gracefully and update the datastore properly, calling the datastore methods that do the magic they do.

Well, the Item won't be available as $vbulletin->name if it wasn't in $specialtemplates before calling global.php.
And as Plugins are being loaded out of the Datastore it can't be added through Hooks. That's the whole point of this Thread:
We've got a "Chicken and Egg" Problem here :)
As I already said, your suggestion works fin - when the default Datastore Class (Database) is being used.
But what's your solution when using other Datastore Classes?


Ah, now there we go.
Sure, that would be possible - with the dabwack of having an additional Query for every Pageload.
And that's what the discusssion here is all about: Which approach adds the smallest Overhead possible?

The Datastore Classes will take care of Updates, that's not a Problem.

<!-- edit: this bit can be ignored, next para solves the solution in my opinion: As far as i can see there will only ever need to be ONE query if the item doesnt exist in the other datastore types' storage. Not fully grasping what happens with them, i suspect thats how it works. (At least thats how it appears to look).

After the single query on the first pageload it is requested on has passed, there will be no more queries. -->

There also wont be any queries if the product builds its datastore item using the class, because the class will deal with the build and will put it into the proper storage.

And as long as the product manages datastore changes properly - it isnt an issue.

How does the Datastore Class (on subsequent Pageloads) know that there are no Entries that need to be cached, and thus does not execute the Query?
How does it know which entries need to be made available as $vbulletin->name?

Looking at the code, there is no precaching features in the other data methods. (The query i posted above predicts what is needed via the new table, and the datasore item does not need to be added to $specialtemplates because of the conditions).

It will indeed make another call to the other datastore storage, but as i understand it (and i could be wrong), for memcached it will query the storage each time it needs an item regardless of if it is stored in the proper storage method and if it isnt, it will query the datastore table and add it to cache.

For file based, as far as i understand, the entire datastore is loaded into memory. I could be very wrong in this assumption and there may need to be file changes to both datastore classes to make it work properly. It may turn out that for other datastore methods (if we go with the table solution) there needs to be an additional query to the NEWTABLE table.

Im not 100% sure that it will work for them, because I dont really know the mechanics. Ill look into it furthur if I have time today.

Yes, all Data (that was once put and TTL isn't over) will be in memory.
But it won't be available to vBulletin Code, as the Class doesn't recognize the Items if they are not default Items or in $specialtemplates.

So, without knowing how they are stored exactly - how should (Plugin) Code get access?

Now i see what you're asking! :)

-- automerge sux!!! --


Looking through the code, init.php simply calls
The fetch() function of the 3 different datastore classes:

• DB: The fetch function builds a comma delimited list of titles to fetch from the database - the query modification I posted will deal with additional datastore retrieval needs, they will all be automatically registered, but not unserialized.
• MEMCACHED: Memcached stores all datastore items in its storage method once it has seen that item at least once - First time it gets seen, it has to be queried, I believe this is fair. $specialtemplates dictates what is automatically registered and unserialized. There should be no extra noticable overhead to call the fetch method for this class type. It will check memcached's store to see if it exists, and if it doesnt query the db, put it in memcached, and register it. An extra function call is necessary for this class type with the current code.
• FILE: It appears that only certain items determined by the $cachableitems array inside this class are able to be stored efficiently for this method. Any datastore items not mentioned in this array are pulled from the database regardless of it they exist in the datastore_cache.php file. This method seems to be a hybrid of the DB and complete filesystem based methods. Maybe this is an oversight, or by design. Changes need to be made to fetch() to allow function calls in products to fetch datastore items that are actually stored in the filesystem, instead of querying the database.

There is also a stress point using fetch, because this function is not necessary and will cause an extra query when using the default datastore method. Maybe another function is necessary to determin if we really need to call fetch() (user using default method? its already registered!!)

One small almost irrelevant point, it would be nice to be able to specify if the datastore item should be unserialized. In my opinion the best way to deal with that would be to have another parameter for fetch() or register() that the caller tells the function to unserialize, instead of relying on an internal $unserialize array.

Hope I made sense :)

I should also note that I do not write this as an intention to hack vBulletin - Im discussing this as an addition to vBulletin. Maybe not the best forum for it, but Im sure the devs are looking at this thread.

By the time we started this thread RC1 was not yet (or just the day before) released. Jelsoft Dev's have been watching this thread before, and we had the slight hope (very long shot) that something could be done into the production release. But i think Jelsoft is way to far into the release process, to make a change to the system like at now.

In my opinion, worsed case scenario would be a standard solution that all coders could use, so files would only have to be edited once to implement custom datastore items.

Edit: You are encouraged to post a proof of concept here so it will be easier to discuss.

Im sorry, I dont have the time to work on a proof of concept test base for code.

I do not believe that the idea needs to be tested, as whatever solution is presented will appear in vBulletin in a form that the developers believe is the most efficient and secure eventually.

I dont understand your statement "Based on what condition you want to remove this?" and where it fits with what im saying?

Looks like that was somehow copied from a reply to another thread. Don't know how that happened. Removed that part.

What I have been doing for my forums and my private hacks is call the extra datastore items from fetch() and not worry about the extra query at this time.

In my opinion people are too query-mad. 1 or 2 extra very light weight queries are no fuss in the shortterm for most boards.

Once a solution appears, the queries go away.

--

On that note, i dont believe there necessarily needs to be any kind of single method of doing it if this functionality is eventually going to get added to vBulletin - the way we do it wont match the way they do it, so why have hacks that create incompatibilities like that?

Maybe we could at least get an official word on their stance, then we would at least know what our position is.

I am sure they havent decided yet..cause it sure is taking much longer than before to come out with RC3 or Gold.

~Curt

To be perfectly honest, I believe they already know what they are going to do. A company like this doesnt just make decisions as things happen, they make them as problems become known.

An official word on where datastore caching for products is at will help this discussion of the need to set a standard way to do it until such time as they do their thing.

If it wont be added until 3.5.0+1 then there is every chance hackers will want to agree on a standard way to do things as Marco suggests. If it will be available before 3.5.0 becomes gold, I see no issue with waiting for the feature to be added instead of working our own solution out.

I've been thinking about a way around the problem in the meantime until Jelsoft do something to solve it, but how about using a plugin to store the extra "datastore" items a product needs to add?

Add a global_start plugin holding the array or variables you need.

Let your product update that global_start plugin and rebuild the plugin datastore items.

Could very well be an acceptable solution in the meantime?

Ive just hit a major major snag with my code using
to fetch data.

It appears in global_start it occurs too late in the process and overwrites certain datastore objects (because it requeries all default datastore objects and fetches them as well) causing the time offset to be incorrect.

How do you guys currently retrieve datastore objects? Just a query? File edits?

File Edits.
If you do it in config.php it will be a one-time edit :)

config.php - interesting.

What does the variable look like?

At the moment im happy to introduce a query for this purpose until a proper solution is made.

Here's what I use the Kirby came up with and it works excellently by the way. ;)

Also, you don't need an if THIS_SCRIPT with it doing it this way. It works globablly. I have one for forumdisplay and 2 for forumhome in there and they all work fine. ;)

I added the little description to make it look more uniformly. ;)

If you put that into config.php, you will only be able to load datastore objects for every page.

THIS_SCRIPT checks allow you to load datastore objects for a per-page basis.

Although this is a nice and simple solution, it does load everything for every page as mentioned. Also i wouldn't be so happy if all hacks where coded this way, it would be a huge config.php loaded on every page.

Honestly, class_core.php is way bigger then config.php would ever get even if every Hack added Items there ;)

But as mentioned, it should be wrapped in if (THIS_SCRIPT == ... )

Well, since I only have 3 of them, that is not a big issue, but yes, you could do it that way. This way, if I want to use it board-wide, I can. ;)

I am using one extra query to load my datastore items.

Just use global_start hook to do a single query which will load the items you need. I am currently happy with one extra light weight query.