CODERS PLEASE DISCUSS!!! Pre-Datastore modifications with XML-files (UPDATE 28-7)

[merk]

Originally Posted by KirbyDE

How does the Datastore Class (on subsequent Pageloads) know that there are no Entries that need to be cached, and thus does not execute the Query?
How does it know which entries need to be made available as $vbulletin->name?

Looking at the code, there is no precaching features in the other data methods. (The query i posted above predicts what is needed via the new table, and the datasore item does not need to be added to $specialtemplates because of the conditions).

It will indeed make another call to the other datastore storage, but as i understand it (and i could be wrong), for memcached it will query the storage each time it needs an item regardless of if it is stored in the proper storage method and if it isnt, it will query the datastore table and add it to cache.

For file based, as far as i understand, the entire datastore is loaded into memory. I could be very wrong in this assumption and there may need to be file changes to both datastore classes to make it work properly. It may turn out that for other datastore methods (if we go with the table solution) there needs to be an additional query to the NEWTABLE table.

Im not 100% sure that it will work for them, because I dont really know the mechanics. Ill look into it furthur if I have time today.

[Andreas]

Yes, all Data (that was once put and TTL isn't over) will be in memory.
But it won't be available to vBulletin Code, as the Class doesn't recognize the Items if they are not default Items or in $specialtemplates.

So, without knowing how they are stored exactly - how should (Plugin) Code get access?

[merk]

Now i see what you're asking!

-- automerge sux!!! --


Looking through the code, init.php simply calls

PHP Code:

$vbulletin->datastore->fetch($specialtemplates); 


The fetch() function of the 3 different datastore classes:

• DB: The fetch function builds a comma delimited list of titles to fetch from the database - the query modification I posted will deal with additional datastore retrieval needs, they will all be automatically registered, but not unserialized.
• MEMCACHED: Memcached stores all datastore items in its storage method once it has seen that item at least once - First time it gets seen, it has to be queried, I believe this is fair. $specialtemplates dictates what is automatically registered and unserialized. There should be no extra noticable overhead to call the fetch method for this class type. It will check memcached's store to see if it exists, and if it doesnt query the db, put it in memcached, and register it. An extra function call is necessary for this class type with the current code.
• FILE: It appears that only certain items determined by the $cachableitems array inside this class are able to be stored efficiently for this method. Any datastore items not mentioned in this array are pulled from the database regardless of it they exist in the datastore_cache.php file. This method seems to be a hybrid of the DB and complete filesystem based methods. Maybe this is an oversight, or by design. Changes need to be made to fetch() to allow function calls in products to fetch datastore items that are actually stored in the filesystem, instead of querying the database.

There is also a stress point using fetch, because this function is not necessary and will cause an extra query when using the default datastore method. Maybe another function is necessary to determin if we really need to call fetch() (user using default method? its already registered!!)

One small almost irrelevant point, it would be nice to be able to specify if the datastore item should be unserialized. In my opinion the best way to deal with that would be to have another parameter for fetch() or register() that the caller tells the function to unserialize, instead of relying on an internal $unserialize array.

Hope I made sense

I should also note that I do not write this as an intention to hack vBulletin - Im discussing this as an addition to vBulletin. Maybe not the best forum for it, but Im sure the devs are looking at this thread.

[Marco van Herwaarden]

By the time we started this thread RC1 was not yet (or just the day before) released. Jelsoft Dev's have been watching this thread before, and we had the slight hope (very long shot) that something could be done into the production release. But i think Jelsoft is way to far into the release process, to make a change to the system like at now.

In my opinion, worsed case scenario would be a standard solution that all coders could use, so files would only have to be edited once to implement custom datastore items.

Edit: You are encouraged to post a proof of concept here so it will be easier to discuss.

[merk]

Im sorry, I dont have the time to work on a proof of concept test base for code.

I do not believe that the idea needs to be tested, as whatever solution is presented will appear in vBulletin in a form that the developers believe is the most efficient and secure eventually.

I dont understand your statement "Based on what condition you want to remove this?" and where it fits with what im saying?

[Marco van Herwaarden]

Originally Posted by merk

I dont understand your statement "Based on what condition you want to remove this?" and where it fits with what im saying?

Looks like that was somehow copied from a reply to another thread. Don't know how that happened. Removed that part.

[merk]

What I have been doing for my forums and my private hacks is call the extra datastore items from fetch() and not worry about the extra query at this time.

In my opinion people are too query-mad. 1 or 2 extra very light weight queries are no fuss in the shortterm for most boards.

Once a solution appears, the queries go away.

--

On that note, i dont believe there necessarily needs to be any kind of single method of doing it if this functionality is eventually going to get added to vBulletin - the way we do it wont match the way they do it, so why have hacks that create incompatibilities like that?

Maybe we could at least get an official word on their stance, then we would at least know what our position is.

[bigcurt]

I am sure they havent decided yet..cause it sure is taking much longer than before to come out with RC3 or Gold.

~Curt

[merk]

Originally Posted by bigcurt

I am sure they havent decided yet..cause it sure is taking much longer than before to come out with RC3 or Gold.



~Curt

To be perfectly honest, I believe they already know what they are going to do. A company like this doesnt just make decisions as things happen, they make them as problems become known.

An official word on where datastore caching for products is at will help this discussion of the need to set a standard way to do it until such time as they do their thing.

If it wont be added until 3.5.0+1 then there is every chance hackers will want to agree on a standard way to do things as Marco suggests. If it will be available before 3.5.0 becomes gold, I see no issue with waiting for the feature to be added instead of working our own solution out.

[merk]

I've been thinking about a way around the problem in the meantime until Jelsoft do something to solve it, but how about using a plugin to store the extra "datastore" items a product needs to add?

Add a global_start plugin holding the array or variables you need.

Let your product update that global_start plugin and rebuild the plugin datastore items.

Could very well be an acceptable solution in the meantime?