Log in

View Full Version : CODERS PLEASE DISCUSS!!! Pre-Datastore modifications with XML-files (UPDATE 28-7)


Marco van Herwaarden
07-27-2005, 04:14 PM
Coding in this thread should be considered a Proof of Concept, not to be used outside testboards!!!!!

Alternative 2 presented

Ok now i have got your attention, please read the following. I really want this to become a combined effort of the vbulletin.org community. If we can pull this of in a good way we could try to either convince Jelsoft that this should be standard (i doubt they will do this in 3.5, but worth a try), or we could use it as a standard extension that can be used by many coders. Or everybody will just shoot me for this idea. :D

We got all these nice plugins etc, but there is still something that can not be done without file edits:
Adding datastore items ($specialtemplates)

Before it also seemed impossible to touch prefetching templates ($globaltemplates & $actiontemplates) or new Phrasegroups ($phrasegroups), although i think most of these could be done with a plugin nowadays

The datastore seems untouchable because it gets loaded before the plugin system gets active.

Alternative 1

See file: Pre-Datastore using XML-files - v1.00.zip (https://vborg.vbsupport.ru/attachment.php?attachmentid=32464&stc=1)

I made 1 modification to ./includes/init.php that could solve this problem. What it does is give you the chance to create a XML-file for your hack that will be loaded before the datastore gets build.

To use this a 1 time modfication to init.php would be needed (unless Jelsoft wants to use it in the standard product), that could be shared by all hacks.

It let's you add in a controled way entries to the following vars:
- $phrasegroups
- $specialtemplates
- $globaltemplates
- $actiontemplates (not working in this version, would have to do some small changes for it)

I have been testing it on my localhost and it doesn't add much to the pageloads (important, it would be executed on each page load).

Please all coders have a look at this modification and give feedback (positive/negative or improvements)

In ./includes/init.php find:
$db->timer_start('Datastore Setup');
}

Add after:
// ################################################## ###########################
// Start Hack: Pre-Datastore Plugin (MarcoH64)
// Optional TODO: Check $xml['product'] is active
// Not sure if this would be needed in this situation, it can't harm much if it is always active, unless a lot of removed plugins
// But in that case loading might get slow also
// Reading the product table for now to see if active, this extra query will slow down things, so maybe just remove
// We might also want to store in teh $vbulletin object, so it won't need to be queried later again
// Retrieve active products
$actproducts = $db->query_read("SELECT productid FROM " . TABLE_PREFIX . "product WHERE active = 1");
$actprods = array();
while ($actproduct = $db->fetch_array($actproducts))
{
$actprods[] = $actproduct['productid'];
}
unset($actproducts);
// / Optional TODO: Check $xml['product'] is active
require_once(DIR . '/includes/class_xml.php');
// Maybe seperated directories for the different types of xml-files could speed up processing of the
// list of files. List might get long with a lot of hacks installed. Example:
// ./includes/xml/cpnav <- Only ACP navigation
// ./includes/xml/preds <- Only Pre-Datastore
$handle = opendir(DIR . '/includes/xml/');
while (($file = readdir($handle)) !== false)
{
if (!preg_match('#^preds_(.*).xml$#i', $file, $matches))
{
continue;
}
$preds_file = $matches[1];
$xmlobj = new XMLparser(false, DIR . "/includes/xml/$file");
$xml =& $xmlobj->parse();
// Optional TODO: Check $xml['product'] is active
// Always allow product 'vbulletin' :))))))))))))
if ($xml['product'] != 'vbulletin'
AND !in_array($xml['product'], $actprods))
{
continue;
}
// / Optional TODO: Check $xml['product'] is active

if (!is_array($xml['predsgroup'][0]))
{
$xml['predsgroup'] = array($xml['predsgroup']);
}

foreach ($xml['predsgroup'] AS $predsgroup)
{
// Check if we should handle this entry for the current script
// If the entry is marked with script=global, then always include
// If no script is given, this entry is poorly written and should be distrusted and skipped
// If vb_area is not set in xml-file, or no match, skip. This is done to protect from running while in ACP if lazy coders ommit the area
if ((trim($predsgroup['script']) != 'global' AND trim($predsgroup['script']) != THIS_SCRIPT)
OR (trim($predsgroup['vb_area']) != VB_AREA))
{
continue;
}

// Handle adding to $specialtemplates
if ($predsgroup['specialtemplates'])
{
if (!is_array($predsgroup['specialtemplates'][0]))
{
$predsgroup['specialtemplates'] = array($predsgroup['specialtemplates']);
}
$specialtemplates = (is_array($specialtemplates) ? array_merge($specialtemplates, $predsgroup['specialtemplates'][0]) : array_merge(array($specialtemplates), $predsgroup['specialtemplates'][0]));
}

// Handle adding to $phrasegroups
if ($predsgroup['phrasegroups'])
{
if (!is_array($predsgroup['phrasegroups'][0]))
{
$predsgroup['phrasegroups'] = array($predsgroup['phrasegroups']);
}
$phrasegroups = (is_array($phrasegroups) ? array_merge($phrasegroups, $predsgroup['phrasegroups'][0]) : array_merge(array($phrasegroups), $predsgroup['phrasegroups'][0]));
}

// Handle adding to $globaltemplates
if ($predsgroup['globaltemplates'])
{
if (!is_array($predsgroup['globaltemplates'][0]))
{
$predsgroup['globaltemplates'] = array($predsgroup['globaltemplates']);
}
$globaltemplates = (is_array($globaltemplates ) ? array_merge($globaltemplates , $predsgroup['globaltemplates'][0]) : array_merge(array($globaltemplates ), $predsgroup['globaltemplates'][0]));
}

// Handle adding to $actiontemplates
if ($predsgroup['actiontemplates'])
{
if (!is_array($predsgroup['actiontemplates'][0]))
{
$predsgroup['actiontemplates'] = array($predsgroup['actiontemplates']);
}
$actiontemplates = (is_array($actiontemplates) ? array_merge($actiontemplates, $predsgroup['actiontemplates'][0]) : array_merge(array($actiontemplates), $predsgroup['actiontemplates'][0]));
}
}
$xmlobj = null;
unset($xml);
}
// End Hack: Pre-Datastore Plugin (MarcoH64)
// ################################################## ###########################

Save and upload init.php.

That is all.

How to use this?

As said before you can now use a XML-file uploaded to the ./includes/xml' directory to control things.

XML-File naming: ./includes/xml/preds_<productid>.xml
Replace the <productid> preferable with the productid you assigned to your hack, but any other unique name should also work.

Now the content of the file:
<?xml version="1.0" encoding="ISO-8859-1"?>
<predsgroups product="productid">
<predsgroup script="global" vb_area="AdminCP">
<phrasegroups>phrasegroup</phrasegroups>
<specialtemplates>datastoreitem</specialtemplates>
<globaltemplates>globaltemplate</globaltemplates>
<actiontemplates>actiontemplate</actiontemplates>
</predsgroup>
<predsgroup script="showthread" vb_area="Forum">
<phrasegroups>phrasegroup</phrasegroups>
<specialtemplates>datastoreitem</specialtemplates>
<globaltemplates>globaltemplate</globaltemplates>
<actiontemplates>actiontemplate</actiontemplates>
</predsgroup>
</predsgroups>


Note: Like said before actiontemplates don't work yet, so don't even try to use them.

- You must set a product! (if your product is not enabled, code will not run from this file.
- script must contain the name of the file for which this group should be loaded, or 'global' to always load.
- vb_area MUST be set. Default vB has 1 areas 'Forum' and 'AdminCP'
- You can have as many predsgroup's as you need.
- Within each group you can have as many lines as you need. Only the above tags are supported, all other will be ignored.


Please all have a look at this 'proof of concept' and give feedback.

If we think this would be a nice way of standarization with only once a code edit (unless.....) then i will release this to use for all hacks released here on vb.org, if the coder wants to use it.


Alternative 2

See for details file: Proof of Concept Pre-Datastore for plugins - Alternative 2 Version 1.00.zip

Description:
Proof of Concept Pre-Datastore for plugins - Alternative 2 Version 1.00
This code is part of a proof of concept discussion at vbulletin.org. Goal of this discussion is to create a way
to add new datastore items in hacks without having to modify source code for each hack.
Unless Jelsoft decide to incorporate one of the presented alternatives this goal will never be met. At least there would be needed
a one time code modification for all hacks:
Thread: https://vborg.vbsupport.ru/showthread.php?t=93007
Author: MarcoH64
This code might NOT be used as a whole or a part without authors permission.
*** THIS CODE IS STRICTLY TO BE USED FOR CONCEPTUAL TESTING, AND NOT TO BE USED OUTSIDE A TESTBOARD ***
================================================== ================================================== =
Alternative 2:
Add a new table that will store information of datastore items needed for any script.
Merge the info of that table with the standard vB datastore items that are retrieved in unmodified code.
The idea of this alternative was presented by different members on vb.org and vb.com.
Pro in this solution:
- No need to parse XML-files for each pageload (compaired to Alternative 1)
- Should probably be faster because of this.
- Table can be loaded together with the default Product XML-file
Con's in this alternative:
- Will also execute if Plugin System is globally disabled (chicken and egg problem, or extra queries needed)
- A lot more file modifications needed
- Script to maintain the information stored in the table not written (yet?).
Should anyway only be available in debug mode?
- Need to create a new table
- Query to use when loading the datastore on each page load might be slowed down considerably. This should be tested
in a representable environment. Same goes for alternative 1.
- This alternative only handles new datastore items, not template caching like alternative 1

deathemperor
07-27-2005, 04:19 PM
exactly what I was thinking of how I was supposed to deal with uncached templates and things listed above.

A very nice hack, Macro. I hope vb.com will consider this.

Marco van Herwaarden
07-27-2005, 04:24 PM
Let's make it perfect as a combined effort, and maybe we can also as a combined effort try to get it into standard vB.

Andreas
07-27-2005, 04:34 PM
Hmm, it adds 1 Query to every Page being loaded, and parsing several XML Files for every Page also adds Overhead.
Global and Action Templates can already be added through Plugins, not sure if this is possible for Phrasegroups too - but I think Phrasegroups might not be an issue at all, as those moste likely would be used in additional Files anyway.

So what's left is Datastore.

I posted a somewhat similar idea @ vbulletin.com some time ago:
http://www.vbulletin.com/forum/showthread.php?t=146141

IMHO, this should not add any Overhead at all.

Or, to keep your idea:
As it requires an additional Query, what about the following:

A Table requirements with colums

scriptid
globaltemplates
actiontemplates
phrasetypes
specialtemplates
product
area


init.php would then query this table for the current scriptid/area, joined on products that are active.
Then merge the results with the current values and continue execution.

Adrian Schneider
07-27-2005, 04:54 PM
Thanks Macro, I will test this out today, as I was having this problem earlier. :)

Marco van Herwaarden
07-27-2005, 05:05 PM
Hmm, it adds 1 Query to every Page being loaded,
Yes that is also why i wrote in my comment that we should discuss if testing if a product is active would be something that is really needed.

Keep in mind please that i also want this to be a disccusion, and if we all agree that this test is not worth the extra query, we could remove it easy.
As it requires an additional Query, what about the following:

A Table requirements with colums
scriptid
globaltemplates
actiontemplates
phrasetypes
specialtemplates
product
area

init.php would then query this table for the current scriptid/area, joined on products that are active.
Then merge the results with the current values and continue execution.
That would be an even more heavy query i guess.

Another alternative to the solution i have choosen here would be to load all this data from the XML-files (should be loaded on install of a product then) into a table, and then add this data serialized (hmm now i remember i wanted to add support for serialized datastore items, will do this tomorrow) to the datastore. This new record in the datastore shoudl always be queried when loading datastore items, and the load routine for the datastore could then dynamicly add those items to the datastore. But this would also mean another query (first query to retrieve this new record, then another to retrieve additional information, or retrieve all records from datastore, but only store the wanted ones into the $vbuleltin->datastore)

Marco van Herwaarden
07-27-2005, 05:06 PM
Thanks Macro, I will test this out today, as I was having this problem earlier
I ran into the datastore problem myself when working on the vB.org Hack Database. And i just don't want to be editing 10 files, only to add the datastore item and do all other things with plugins.

amykhar
07-27-2005, 05:09 PM
I haven't been able to get the phrasegroup working in a plugin and I need it to work for two existing files. I ended up just making the phrases global phrases, but that's not optimal.

Amy

Andreas
07-27-2005, 05:15 PM
That would be an even more heavy query i guess.
Yes. But as the Tables won't be big it should not make much of a difference; at least it should be faster then opening and parsing a bunch of Files.

Hmm, it might be even possible to achieve smth. without additional Queries:

Have all requirements (eg. Phrases, Global and Action Templates - except Datastore Items) as a serialized DS Item that is always being loaded.
Combined with the Idea in the linked Thread above, it should not require any further Queries.


or retrieve all records from datastore, but only store the wanted ones into the $vbuleltin->datastore)

Wouldn't that require way more memory/database bandwidth?

Marco van Herwaarden
07-27-2005, 05:18 PM
You mean you want to load ALL phrases and templates into the datastore???

Edit: PS Maybe you should update your signature :D

Marco van Herwaarden
07-27-2005, 05:20 PM
Wouldn't that require way more memory/database bandwidth?
Not really, bandwidth is not really an issue, since it is just a server process, and the datastore table will never have hundreds of records.

Edit: Memory used to keep the datastore in memory for later processing by scripts is more an issue then the memory used while retrieving everything.

I will try to make some alternatives tomorrow.

Andreas
07-27-2005, 05:22 PM
What I meat was a serialized array with the Requirements for each Product.
This item should always be queried then, the Load Routine should extract the currently needed Data (identified by scriptid/area) and merge it with the existing $globaltemplates, $actiontemplates, $phrasegroups and continue execution - just like it does with the XML Files idea.

Marco van Herwaarden
07-27-2005, 05:28 PM
I will have a sleep over this all, not sure yet that what you are suggesting could be done with less performance issues then the solution i posted above, but maybe we should write out all possible solutions and find some big boards who want to be testing it and marking execution times. On my localost, the above solution (including the query) added 0.01 second to the page load.

PS I made a reply at your thread on vb.com, asking others to join us here in the discussion.

Andreas
07-27-2005, 05:32 PM
I am not sure either :)
But in Theory a Query (if it's not heavy) should be faster then File Operations and Parsing.

Link14716
07-27-2005, 05:46 PM
I say it should load all the entries from the XML files regaurdless of whether the product is enabled or not to save the query.

I say Kirby's original idea at vB.com sounds good with a few tweaks for datastore stuff.

Marco van Herwaarden
07-27-2005, 07:58 PM
I will also have a go at implementing that idea tomorrow, unless someone else volunteers. In the end i hope that we come together with the best solution and try to persuade Jelosoft in integrating it, or we could use it here for all coders who want to use it.

tamarian
07-27-2005, 08:01 PM
Great idea, here are some comments

1. We really should find something to eliminate the need for an extra query and an xml parse for each page view. Large sites cannot afford this. Of course smaller sites can use this, but hack authors would still require filed edits, so their hacks don't slow down large sites.

2. Since there are file edits anyway, why not edit global.php instead. I think some re-arrangement can help, such as moving init_language to below a hook (can't see any side effects, but I could be wrong). This, I assume will solve the phrasegroup issue?

3. What's left are templates. But since we can already modify global templates with a plugin, and if it uses an if statement to check for the script being used before merging the hack's template, then we're done (except for special templates)

But, let's hope vB Dev's surprise us before gold, and we don't have to mock things up :) I noticed Scott's reply to Kirbey's thread, so I am optimistic.

Marco van Herwaarden
07-27-2005, 08:21 PM
Thanks for the input tamarian, will have a good sleep over this and try to find some alternatives tomorrow.

Didn't notice the reply at vb.com yet, but will go have a look.

To all other coders........input please.

Chris M
07-27-2005, 09:31 PM
It sounds like a great addition, but I would like to see what the Dev's take on this :)

Satan

merk
07-27-2005, 11:26 PM
I thought if you had another table all that would be required is to modify the datastore query that is already present to join another table with more conditions for extra things to pull

ie


SELECT *
FROM datastore
JOIN NEW TABLE
WHERE title IN(blah) OR
NEW TABLE CONDITIONS (IF title=title and newtable.active)

Marco van Herwaarden
07-28-2005, 02:59 PM
Second Alternative presented.

This one uses a table that gets merged during loading of the datastore rows.
Data is released inside the regular Product XML file that you use for your hack.

See newly attached file for details.

Again, please comment!!!!

merk
07-29-2005, 12:26 AM
Con's in this alternative:
- Will also execute if Plugin System is globally disabled (chicken and egg problem, or extra queries needed)
- A lot more file modifications needed
- Script to maintain the information stored in the table not written (yet?).
Should anyway only be available in debug mode?
- Need to create a new table
- Query to use when loading the datastore on each page load might be slowed down considerably. This should be tested
in a representable environment. Same goes for alternative 1.
- This alternative only handles new datastore items, not template caching like alternative 1

If the plugin system is disabled, it could set a flag in the system (somehow) to also not load the extra datastore items, though it probably wont work because vboptions is in the datastore.

Not having looked at your proof of concept, I would have thought the only file modification needed would be in the datastore class to modify the query. It shouldnt be necessary on each pageload to do anything else..

I dont see creating another table as an issue - the more tables you have to deal with things the better! :) - I also dont believe that a query joined to this table would be slower than reading an XML file from the filesystem.

Template caching can already be achieved with a hook..?

deathemperor
07-29-2005, 01:12 AM
Global and Action Templates can already be added through Plugins
Sorry if this goes off topic, but can you share your way to code this ?

Marco van Herwaarden
07-29-2005, 03:32 AM
If the plugin system is disabled, it could set a flag in the system (somehow) to also not load the extra datastore items, though it probably wont work because vboptions is in the datastore.I don't think that would chang a lot, since it would have to be stored somehow, it would also have to be retrieved, resulting in an extra query. (Not that an extra well designed query should always be such a big issue).
Not having looked at your proof of concept, I would have thought the only file modification needed would be in the datastore class to modify the query. It shouldnt be necessary on each pageload to do anything else..To implement it, only 1 modification to the datastore class is made. Ofcourse this added code must be executed on each pageload, how you would want to retrieve the info otherwise?

The other modifications are management code (loading the table with the product XML-File, disabling the entry when the product is disabled, etc.)
I dont see creating another table as an issue - the more tables you have to deal with things the better! - I also dont believe that a query joined to this table would be slower than reading an XML file from the filesystem.Creating another table is not an issue if Jelsoft is doing it, it is something to consider when a Third Party (us) is making changes to the database design.
Template caching can already be achieved with a hook..?
I am taking the word of others on this, not tried it myself, but this was the beauty of Alternative 1 in my eyes. We could ofcourse always create something similar to the datastore hack for this. (Could be stored in datastore, so no extra query would be needed.)

Andreas
07-29-2005, 08:25 AM
Sorry if this goes off topic, but can you share your way to code this ?
Hook cache_templates

Andreas
07-29-2005, 08:35 AM
Con's in this alternative:
- Will also execute if Plugin System is globally disabled (chicken and egg problem, or extra queries needed)

Uhh, yes - that is a Problem. Hmm, thinking about that.


- A lot more file modifications needed

Haven't really implemented it yet, but in theory it should be just a few lines in init.php?


- This alternative only handles new datastore items, not template caching like alternative 1
Idea: Seriaized DS Entry that holds Phrase/Template Requirements for every Script. This entry would have to be loaded always, the currently required Data extracted and merged with $phraseggroups etc.

Marco van Herwaarden
07-29-2005, 08:48 AM
Haven't really implemented it yet, but in theory it should be just a few lines in init.php?Already answered by:
To implement it, only 1 modification to the datastore class is made. Ofcourse this added code must be executed on each pageload, how you would want to retrieve the info otherwise?

The other modifications are management code (loading the table with the product XML-File, disabling the entry when the product is disabled, etc.)

Quote:

Idea: Seriaized DS Entry that holds Phrase/Template Requirements for every Script. This entry would have to be loaded always, the currently required Data extracted and merged with $phraseggroups etc.

Since our primary focus was on datastore and templates/phrases can already be done by plugins, this is not coded into Alternative 2. In alternative 1, i was able to easily catch all 4 arrays, within the same structure and edit location.

To implement phrases/templates into Alt. 2, completly different code would be needed compaired with the datastore solution (because the problem is different - adding things prior to having the datastore available - versus - adding things after we have the datastore). What they do have in common is that we could use the same table to store the info, with some slight modifications. I even started with columns in the table to support this, but i removed them later since i didn't plan to implement it.

PS We are being watched by Jelsoft on the progress in this thread (i think :D) and there might be a very slight chance that Jelsoft might adopt an idea (which would be the best solution for the community i think).

Marco van Herwaarden
07-29-2005, 08:52 PM
I am a bit disappointed at the lack of feedback and discussion in this thread from most coders.

Paul M
08-05-2005, 08:15 PM
I am a bit disappointed at the lack of feedback and discussion in this thread from most coders.Probably because having read it, I'm somewhat confused as to what the original problem is/was and what exactly you are trying to achieve. Perhaps I'll have another read after my holiday.

(and unrelated - but long thread titles with tons of CAPITALS in them are ones I tend to avoid ....)

Revan
08-05-2005, 08:36 PM
So let me see if I understood this correctly - you want to be able to add new phrasegroups/actiontemplates to existing vB pages?

My suggestion:

AdminCP:
New file: Import/Export/Editing/Deletion of an XML file record containing list of templates/phrases required for each mod. A dupe of the plugin system, so to speak, only it contains a possiblity to amend to every single instance of the $xxxtemplates arrays, including subarrays.
Every time something is added/edited/removed from this list, a flatfile (similar to the flatfile DS cache option in vB) is created. Obviously it does array searches to make sure no duplicates are present (although they wouldn't do much harm).

Init.php:
Find
require_once(CWD . '/includes/class_core.php');
Add below
require_once(CWD . '/includes/datastore_precache.php');
From there, different parts of init.php can use variables from this file (again, similar to the datastore_cache.php) to do whatever it is you want it to do, such as array_merge($precache_templates, $specialtemplates); etc.


A simple require_once() won't bring any huge load onto the server, it won't be parsing anything and it won't be querying anything.

Link14716
08-05-2005, 08:52 PM
The problem is then you have to CHMOD to install plugins.

I still say the new table/query modification is the best for datastore, and hooks can take care or everything else.

Revan
08-05-2005, 08:57 PM
So what if you have to CHMOD 1 file once?
You don't have to CHMOD the current datastore_cache every time you rebuild the vBOptions if you use the flatfile method, do you?
And even IF you would have to, it's still more efficient than adding queries or doing advanced parsing.

Erwin
08-06-2005, 01:44 AM
So what if you have to CHMOD 1 file once?
You don't have to CHMOD the current datastore_cache every time you rebuild the vBOptions if you use the flatfile method, do you?
And even IF you would have to, it's still more efficient than adding queries or doing advanced parsing.

That's what I've done in the past on my site. It does work. But would defeat the purpose of the concept of plugins.

Marco van Herwaarden
08-06-2005, 05:45 AM
Nice to see there is some life in this topic again.

Let me just try to get the original problem clear again:
- You can do a lot without code changes with 3.5, there is however 1 thing you can not: If your hack adds a new entry to the datastore, you will have to edit each file to add your datastore item to the list of items to be fetched. You can not use hooks, because they are only loaded after the datastore is read.
- If we can easily without extra effort also add a simple way to do the same for template caching, then do so. It would reduce the number of plugins needed for your hack.

The original idea was to come with a solution that Jelsoft even might consider implementing, but since we are already on RC2, i doubt that will happen. Second best would be a standarized way of handling this as a community, so that whatever hack you use, file edits would only have to be made once.

Revan
08-06-2005, 10:27 AM
That's what I've done in the past on my site. It does work. But would defeat the purpose of the concept of plugins.How does it defeat the purpose of the concept of plugins?
If the hack is extensive enough to require datastore plugging into, it will also require file upload, so CHMODding a file wouldn't be hassle.
I don't really know, so DO you have to re-CHMOD the datastore_cache.php each time you update vBOptions?
If you don't, I don't see any reason why this should be any different.

Nice to see there is some life in this topic again.

Let me just try to get the original problem clear again:
- You can do a lot without code changes with 3.5, there is however 1 thing you can not: If your hack adds a new entry to the datastore, you will have to edit each file to add your datastore item to the list of items to be fetched. You can not use hooks, because they are only loaded after the datastore is read.
- If we can easily without extra effort also add a simple way to do the same for template caching, then do so. It would reduce the number of plugins needed for your hack.

The original idea was to come with a solution that Jelsoft even might consider implementing, but since we are already on RC2, i doubt that will happen. Second best would be a standarized way of handling this as a community, so that whatever hack you use, file edits would only have to be made once.If I had known this would concern me I would have replied earlier, XD
For my RPG I require a cache, but I just created my own table (rpg_cache, datastore table duplicate) and query this whenever I call $RPG->init(). I never really wanted to plug into the datastore table, because I wanted my variables to be $RPG->item not $vbulletin->item without too much hassle :p
Im not saying this to say "that's the solution", fyi.

The easiest solution would just be to hassle Jelsoft into adding a hook or two to enable us to do what we are trying to do.
If we have to come up with an uniform way of doing this, however, I believe my suggestion's downsides are far smaller than the disadvantages of adding an extra query to pages who won't even need this new field, and parsing the XML file on the fly (Imagine what would happen if the user has 15 hacks that require datastore modification, *shudders*).

Marco van Herwaarden
08-06-2005, 11:06 AM
This is why i opened the discussion, to find the best together. Your way of adding your own table for the rpg hack works, but will add another query also. If many hacks do this, this will mean many queries. And using custom datastore items in your own new script is not a problem. It is using them in plugins into jelsoft scripts that require code edits.

Andreas
08-06-2005, 12:32 PM
@Revan
Not every Hack that does require a DS item necessarily has to come with it's own Files.
Furthermore, it's impossible to intruduce Hooks to handle the Datastore Problem, as Plugins itself are being loaded from the Datastore.
My idea does not introduce a new Query, it just changes an existing one to have a JOIN and 1 more Condition.
As the Tables that would be used are pretty small, this should hardly cause any Overhead.

The idea of keeping a var_dump()ed array in a File is also nice, as it does not cause parsing or Query Overhead.
vBulletin itself does so, if you use the Class for a File-Based Datastore, so I think the CHMOD issue isn't a big one, as this would have to be done only once.
Hmm, Safe Mode and open_basedir restrictions could cause Problems here though.

Revan
08-06-2005, 05:12 PM
This is why i opened the discussion, to find the best together. Your way of adding your own table for the rpg hack works, but will add another query also. If many hacks do this, this will mean many queries. And using custom datastore items in your own new script is not a problem. It is using them in plugins into jelsoft scripts that require code edits.How can I make a custom class recognise custom DS items then?
@Revan
Not every Hack that does require a DS item necessarily has to come with it's own Files.
Furthermore, it's impossible to intruduce Hooks to handle the Datastore Problem, as Plugins itself are being loaded from the Datastore.Oh XD
The idea of keeping a var_dump()ed array in a File is also nice, as it does not cause parsing or Query Overhead.
vBulletin itself does so, if you use the Class for a File-Based Datastore, so I think the CHMOD issue isn't a big one, as this would have to be done only once.
Hmm, Safe Mode and open_basedir restrictions could cause Problems here though.How so? After all, this file could be stored in the same way as the current ds cache file, which obviously work with Safe Mode and open_basedir...

amykhar
08-06-2005, 06:19 PM
How can I make a custom class recognise custom DS items then?

Easy require the special templates.

Revan
08-06-2005, 07:20 PM
Eh? Gonna need abit more detail than that, girl :p

amykhar
08-06-2005, 07:33 PM
Look at the top of just about every vbulletin script. You'll see something like this:

// get special data templates from the datastore
$specialtemplates = array(
'userstats',
'birthdaycache',
'maxloggedin',
'iconcache',
'eventcache',
'mailqueue'
);


That pulls the datastore items into the code you are going to use. All you have to do after that is refer to them.

Andreas
08-06-2005, 08:11 PM
@Marco (and of course also all others interested)
I've re-implemented both Suggestions to cut down File Edits.

Only 1 File Edit (2 Lines) to config.php is required, and it will even survive Upgrades.

What do you think?

For Alternative 2:
Maintaining the Table should be done by Product Install/Uninstall-Codes so we don't have to care for that, except disabling/enabling a Product (which is handled by a Plugin).

I think i'll also implement Revans Idea (var_dump).

phlogiston
08-06-2005, 10:47 PM
I am a bit disappointed at the lack of feedback and discussion in this thread from most coders.So far I've had to add two datastore items and one phrasegroup via edit to global.php On my site that doesn't bother me, but it will discourage me from sharing which is a shame :(

So you definitely have my full support trying to work this one out, but to be honest my knowledge is more patched together from hacking rather than being up to speed to follow the intricacies of trying to optimise how to do this.

Basically: sorry I ain't much practical help but folk are interested & keen to have a good solution to this (Y)

Revan
08-07-2005, 12:21 AM
Look at the top of just about every vbulletin script. You'll see something like this:

// get special data templates from the datastore
$specialtemplates = array(
'userstats',
'birthdaycache',
'maxloggedin',
'iconcache',
'eventcache',
'mailqueue'
);


That pulls the datastore items into the code you are going to use. All you have to do after that is refer to them.Thanks for the lesson, it worked like a charm :)
I even managed to code a workaround for having to prefix my custom datastore items with 'rpg_' (for ease of uninstall, identification etc) to stop my custom class calls to have to have this prefix as well :)

Marco van Herwaarden
08-07-2005, 07:05 AM
So far I've had to add two datastore items and one phrasegroup via edit to global.php
Putting them in global should only be done if they are used everywhere. If they are only used in a few scripts, they should be placed in those scripts, meaning maybe a lot of edits.

Will have a look at your solutions on monday kirby.

Andreas
08-08-2005, 08:42 PM
Monday has almost passed ... at least in my Timezone ;)

Marco van Herwaarden
08-08-2005, 08:48 PM
Yeah i know, and i didn't hae time today. :(
And i even doubt i will have time tomorrow, but will look as fast as i can.

But this don't need to stop other coders to have a look and comment. :D

Andreas
08-16-2005, 09:55 PM
*bump*

?

merk
08-16-2005, 11:09 PM
I really think the solution to this problem could be very simple

Datastore class:


$dataitems = $db->query_read("
SELECT title, data
FROM " . TABLE_PREFIX . "datastore
# LEFT JOIN " . TABLE_PREFIX . "_NEWTABLE_ ON(datastore.title = _NEWTABLE_.title AND \"" . THIS_SCRIPT . "\" IN(_NEWTABLE_.page)
# LEFT JOIN " . TABLE_PREFIX . "product ON (product.productid = _NEWTABLE_.productid)
WHERE title IN ($itemlist)
# OR ( NOT NULL(_NEWTABLE_.title) AND product.active = true )
");

Andreas
08-16-2005, 11:32 PM
That's basically my idea :)

But it does only work for the default Datastore Class - and only for Datastore Items.

merk
08-16-2005, 11:36 PM
But since templates can already be cached via a hook, why does this system need to deal with that?

Similar modifications could be made to other datastore types, though they would probably have to drop the active checking, although, if you set a product as inactive, it could write to the other datastore methods at that point.

The end user doesnt need a script to add rows to the new table, the product xml file should deal with that bit automatically.

Andreas
08-16-2005, 11:50 PM
Phrasegroups for Example currently can't be done with Hooks (at least not for Guests).

I don't really get how you would go about implementing this Idea with other Datastore Classes.
Maybe some example Code?

merk
08-16-2005, 11:59 PM
Looking at the other classes, it shouldnt make any difference.

The other datastore types (file based or memcached) appear to fetch a datastore item if it doesnt exist in their specific storage mode and add it. Meaning it only needs to be uncached for one page load, then its added.

Im not sure how it deals with updated datastore items. I guess the build_datastore() functions deal with that too.

Didnt think of phrasegroups, but it shouldnt be hard to add phrasegroups either - Take a look at the deletion log table and how it is joined into multiple different types of items that could be deleted (posts, threads, pms)

Add a new column to the new table of type


AND _NEWTABLE_.type = 'datastore' / 'phrasegroup'

Andreas
08-17-2005, 12:05 AM
The other datastore types (file based or memcached) appear to fetch a datastore item if it doesnt exist in their specific storage mode and add it. Meaning it only needs to be uncached for one page load, then its added.

Right. But how would the Classes determine if an item has to be cached if it's not in $specialtemplates?
That's the important question :)

merk
08-17-2005, 12:28 AM
Right. But how would the Classes determine if an item has to be cached if it's not in $specialtemplates?
That's the important question :)

Hope it doesnt come across as arrogant - but it doesnt matter. (From what I see from something like the Memcached class, all datastore items that have ever been "queried" are added to the Memcached cache, but a single query happens if it doesnt exist, but only the first time).

From what I see, the $specialtemplates variable is only useful if you want to save queries to the database while using the traditional datastore method.

If the item doesnt exist in the memcached/filestore storage, it will be queried. But only once. After it has been queried, it will be added to the relevant storage.

The biggest issue I see is that what happens when the datastore item is updated in the database. I do not see how it will update the item in the other storage methods - I suspect that it gets updated when you call the build_* functions. I havent looked at that area because I use the database method :)

Andreas
08-17-2005, 09:20 AM
Seems like you don't understand me, or I don't understand you :)

Let's say a Hack needs attachmentcache which isn't in $specialtemplates for the Script.
What should the Plugin do in order to get this?

merk
08-17-2005, 01:17 PM
Seems like you don't understand me, or I don't understand you :)

Let's say a Hack needs attachmentcache which isn't in $specialtemplates for the Script.
What should the Plugin do in order to get this?

Maybe just a crossed wire :)

1) The plugin that accesses a hook should do the standard thing it does to access the datastore item. (IIRC, $vbulletin->name) - however, it may need to be unserialized. I dont see an issue with unserializing in global_start.

2) The product should add a row into the NEWTABLE table with relevant details of the datastore item. Name, Product ID, Scripts that the datastore item should be loaded on (THIS_SCRIPT names) and any other details that are needed - havent fully thought of them.

At this point, the datastore class for DB type will query the datastore table joining the new table into it looking for conditions that match to load extra items (or if it exists in $specialtemplates). See query I posted above.

For the other datastore types, (because there are no queries involved for other types) it should be as simple as querying the datastore table once (an extra query for one page load) to load the datastore item into the other types' storage method.

There is an issue with this in that if the datastore item updates in the datastore, the product will have to deal with it gracefully and update the datastore properly, calling the datastore methods that do the magic they do.

Andreas
08-17-2005, 07:56 PM
1) The plugin that accesses a hook should do the standard thing it does to access the datastore item. (IIRC, $vbulletin->name) - however, it may need to be unserialized. I dont see an issue with unserializing in global_start.

Well, the Item won't be available as $vbulletin->name if it wasn't in $specialtemplates before calling global.php.
And as Plugins are being loaded out of the Datastore it can't be added through Hooks. That's the whole point of this Thread:
We've got a "Chicken and Egg" Problem here :)
As I already said, your suggestion works fin - when the default Datastore Class (Database) is being used.
But what's your solution when using other Datastore Classes?



For the other datastore types, (because there are no queries involved for other types) it should be as simple as querying the datastore table once (an extra query for one page load) to load the datastore item into the other types' storage method.

Ah, now there we go.
Sure, that would be possible - with the dabwack of having an additional Query for every Pageload.
And that's what the discusssion here is all about: Which approach adds the smallest Overhead possible?


There is an issue with this in that if the datastore item updates in the datastore, the product will have to deal with it gracefully and update the datastore properly, calling the datastore methods that do the magic they do.
The Datastore Classes will take care of Updates, that's not a Problem.

merk
08-17-2005, 10:23 PM
Ah, now there we go.
Sure, that would be possible - with the dabwack of having an additional Query for every Pageload.
And that's what the discusssion here is all about: Which approach adds the smallest Overhead possible?


The Datastore Classes will take care of Updates, that's not a Problem.


<!-- edit: this bit can be ignored, next para solves the solution in my opinion: As far as i can see there will only ever need to be ONE query if the item doesnt exist in the other datastore types' storage. Not fully grasping what happens with them, i suspect thats how it works. (At least thats how it appears to look).

After the single query on the first pageload it is requested on has passed, there will be no more queries. -->

There also wont be any queries if the product builds its datastore item using the class, because the class will deal with the build and will put it into the proper storage.

And as long as the product manages datastore changes properly - it isnt an issue.

Andreas
08-17-2005, 11:22 PM
As far as i can see there will only ever need to be ONE query if the item doesnt exist in the other datastore types' storage. Not fully grasping what happens with them, i suspect thats how it works. (At least thats how it appears to look).
How does the Datastore Class (on subsequent Pageloads) know that there are no Entries that need to be cached, and thus does not execute the Query?
How does it know which entries need to be made available as $vbulletin->name?

merk
08-17-2005, 11:27 PM
How does the Datastore Class (on subsequent Pageloads) know that there are no Entries that need to be cached, and thus does not execute the Query?
How does it know which entries need to be made available as $vbulletin->name?

Looking at the code, there is no precaching features in the other data methods. (The query i posted above predicts what is needed via the new table, and the datasore item does not need to be added to $specialtemplates because of the conditions).

It will indeed make another call to the other datastore storage, but as i understand it (and i could be wrong), for memcached it will query the storage each time it needs an item regardless of if it is stored in the proper storage method and if it isnt, it will query the datastore table and add it to cache.

For file based, as far as i understand, the entire datastore is loaded into memory. I could be very wrong in this assumption and there may need to be file changes to both datastore classes to make it work properly. It may turn out that for other datastore methods (if we go with the table solution) there needs to be an additional query to the NEWTABLE table.

Im not 100% sure that it will work for them, because I dont really know the mechanics. Ill look into it furthur if I have time today.

Andreas
08-17-2005, 11:35 PM
Yes, all Data (that was once put and TTL isn't over) will be in memory.
But it won't be available to vBulletin Code, as the Class doesn't recognize the Items if they are not default Items or in $specialtemplates.

So, without knowing how they are stored exactly - how should (Plugin) Code get access?

merk
08-17-2005, 11:39 PM
Now i see what you're asking! :)

-- automerge sux!!! --


Looking through the code, init.php simply calls
$vbulletin->datastore->fetch($specialtemplates);

The fetch() function of the 3 different datastore classes:

DB: The fetch function builds a comma delimited list of titles to fetch from the database - the query modification I posted will deal with additional datastore retrieval needs, they will all be automatically registered, but not unserialized.
MEMCACHED: Memcached stores all datastore items in its storage method once it has seen that item at least once - First time it gets seen, it has to be queried, I believe this is fair. $specialtemplates dictates what is automatically registered and unserialized. There should be no extra noticable overhead to call the fetch method for this class type. It will check memcached's store to see if it exists, and if it doesnt query the db, put it in memcached, and register it. An extra function call is necessary for this class type with the current code.
FILE: It appears that only certain items determined by the $cachableitems array inside this class are able to be stored efficiently for this method. Any datastore items not mentioned in this array are pulled from the database regardless of it they exist in the datastore_cache.php file. This method seems to be a hybrid of the DB and complete filesystem based methods. Maybe this is an oversight, or by design. Changes need to be made to fetch() to allow function calls in products to fetch datastore items that are actually stored in the filesystem, instead of querying the database.
There is also a stress point using fetch, because this function is not necessary and will cause an extra query when using the default datastore method. Maybe another function is necessary to determin if we really need to call fetch() (user using default method? its already registered!!)

One small almost irrelevant point, it would be nice to be able to specify if the datastore item should be unserialized. In my opinion the best way to deal with that would be to have another parameter for fetch() or register() that the caller tells the function to unserialize, instead of relying on an internal $unserialize array.

Hope I made sense :)

I should also note that I do not write this as an intention to hack vBulletin - Im discussing this as an addition to vBulletin. Maybe not the best forum for it, but Im sure the devs are looking at this thread.

Marco van Herwaarden
08-18-2005, 05:46 AM
By the time we started this thread RC1 was not yet (or just the day before) released. Jelsoft Dev's have been watching this thread before, and we had the slight hope (very long shot) that something could be done into the production release. But i think Jelsoft is way to far into the release process, to make a change to the system like at now.

In my opinion, worsed case scenario would be a standard solution that all coders could use, so files would only have to be edited once to implement custom datastore items.

Edit: You are encouraged to post a proof of concept here so it will be easier to discuss.

merk
08-18-2005, 08:44 AM
Im sorry, I dont have the time to work on a proof of concept test base for code.

I do not believe that the idea needs to be tested, as whatever solution is presented will appear in vBulletin in a form that the developers believe is the most efficient and secure eventually.

I dont understand your statement "Based on what condition you want to remove this?" and where it fits with what im saying?

Marco van Herwaarden
08-18-2005, 11:42 AM
I dont understand your statement "Based on what condition you want to remove this?" and where it fits with what im saying?
Looks like that was somehow copied from a reply to another thread. Don't know how that happened. Removed that part.

merk
08-18-2005, 12:18 PM
What I have been doing for my forums and my private hacks is call the extra datastore items from fetch() and not worry about the extra query at this time.

In my opinion people are too query-mad. 1 or 2 extra very light weight queries are no fuss in the shortterm for most boards.

Once a solution appears, the queries go away.

--

On that note, i dont believe there necessarily needs to be any kind of single method of doing it if this functionality is eventually going to get added to vBulletin - the way we do it wont match the way they do it, so why have hacks that create incompatibilities like that?

Maybe we could at least get an official word on their stance, then we would at least know what our position is.

bigcurt
08-18-2005, 03:58 PM
I am sure they havent decided yet..cause it sure is taking much longer than before to come out with RC3 or Gold.

~Curt

merk
08-19-2005, 12:55 AM
I am sure they havent decided yet..cause it sure is taking much longer than before to come out with RC3 or Gold.



~Curt

To be perfectly honest, I believe they already know what they are going to do. A company like this doesnt just make decisions as things happen, they make them as problems become known.

An official word on where datastore caching for products is at will help this discussion of the need to set a standard way to do it until such time as they do their thing.

If it wont be added until 3.5.0+1 then there is every chance hackers will want to agree on a standard way to do things as Marco suggests. If it will be available before 3.5.0 becomes gold, I see no issue with waiting for the feature to be added instead of working our own solution out.

merk
08-21-2005, 11:38 PM
I've been thinking about a way around the problem in the meantime until Jelsoft do something to solve it, but how about using a plugin to store the extra "datastore" items a product needs to add?

Add a global_start plugin holding the array or variables you need.

Let your product update that global_start plugin and rebuild the plugin datastore items.

Could very well be an acceptable solution in the meantime?

merk
09-10-2005, 09:44 PM
Ive just hit a major major snag with my code using
$vbulletin->datastore->fetch(array('menu_left', 'menu_right')); to fetch data.

It appears in global_start it occurs too late in the process and overwrites certain datastore objects (because it requeries all default datastore objects and fetches them as well) causing the time offset to be incorrect.

How do you guys currently retrieve datastore objects? Just a query? File edits?

Andreas
09-10-2005, 11:30 PM
File Edits.
If you do it in config.php it will be a one-time edit :)

merk
09-10-2005, 11:31 PM
config.php - interesting.

What does the variable look like?

At the moment im happy to introduce a query for this purpose until a proper solution is made.

Andreas
09-10-2005, 11:34 PM
if (THIS_SCRIPT == 'whatever')
{
$specialtemplates[] = 'item';
}

Boofo
09-13-2005, 12:28 AM
Here's what I use the Kirby came up with and it works excellently by the way. ;)

Also, you don't need an if THIS_SCRIPT with it doing it this way. It works globablly. I have one for forumdisplay and 2 for forumhome in there and they all work fine. ;)

// ****** SPECIALTEMPLATES *****
// Add any specialtemplates here for any products or mods that use the datastore, to save
// from re-doing file edits on an upgrade or re-install of vBulletin (until they give us a better
// way to do it, anyway). Thanks to KirbyDE for the how-to on doing this.
global $specialtemplates;
$specialtemplates = array_merge(
$specialtemplates, array(
'forumstatscache',
));

I added the little description to make it look more uniformly. ;)

merk
09-13-2005, 12:42 AM
If you put that into config.php, you will only be able to load datastore objects for every page.

THIS_SCRIPT checks allow you to load datastore objects for a per-page basis.

Marco van Herwaarden
09-13-2005, 03:47 AM
Although this is a nice and simple solution, it does load everything for every page as mentioned. Also i wouldn't be so happy if all hacks where coded this way, it would be a huge config.php loaded on every page.

Andreas
09-13-2005, 03:50 AM
Honestly, class_core.php is way bigger then config.php would ever get even if every Hack added Items there ;)

But as mentioned, it should be wrapped in if (THIS_SCRIPT == ... )

Boofo
09-13-2005, 04:09 AM
If you put that into config.php, you will only be able to load datastore objects for every page.

THIS_SCRIPT checks allow you to load datastore objects for a per-page basis.

Well, since I only have 3 of them, that is not a big issue, but yes, you could do it that way. This way, if I want to use it board-wide, I can. ;)

merk
09-13-2005, 11:15 PM
I am using one extra query to load my datastore items.

Just use global_start hook to do a single query which will load the items you need. I am currently happy with one extra light weight query.

Andreas
09-14-2005, 02:13 AM
Here is another approach.

Installation Developer
1) Upload all Files
2) Import product-ds4p.xml
3) At the end (right before ?>) of includes/config.php ADD

require_once(CWD . '/includes/class_datastore_pluginsupport.php');


Installation Enduser
1) Upload class_datastore_pluginsupport.php
2) Import product-ds4p.xml
3) At the end (right before ?>) of includes/config.php ADD

require_once(CWD . '/includes/class_datastore_pluginsupport.php');



Usage Developer
In ACP/Plugin Manager there is a new Entry Datastore Item Manager where you can define the Datastore Requirements for your Product.
When a Plugin requires a Datastore Item that is not being loaded by Default use

$vbulletin->datastore->do_fetch('myitem');

in the Plugin.
That's it.

Usage Enduser
Nothing special, just use Product Management as normal.

Pros
- Supports all storage Methods
- Does take care of Plugin System getting disabled (will not load custom items then) without Overhead
- Easy to use for Developer and Enduser
- Only 1 one-time, one-line File Edit necessary

Cons
- Slightly stronger Query
- New Table added

Brad
09-14-2005, 02:24 AM
Thanks kirby, I'll give this one a try :)

Marco van Herwaarden
09-14-2005, 02:48 AM
Will have a look also today i hope.

merk
09-14-2005, 03:59 AM
Well done kirby, thats basically what I saw the fix being for the requirement.

Andreas
09-18-2005, 01:19 AM
*bump*
Anyone except merk willing to take a look at this?
Sad that I apparently wrote it for nothing :(

Zachariah
09-18-2005, 01:51 AM
*bump*
Anyone except merk willing to take a look at this?
Sad that I apparently wrote it for nothing :(

Never !!! .... I got a hack I can try it with

merk
09-18-2005, 02:39 AM
*bump*
Anyone except merk willing to take a look at this?
Sad that I apparently wrote it for nothing :(

I hope Jelsoft have an eye on this thread because something is needed to allow this to occur, even if they write their own solution to the problem, there are many good ideas in here.

But as ive said before, im not about to modify such a core part of vbulletin when its only 1 more query per page for all my datastore items, until a time comes when it is default.

Its a shame that the default datastore classes dont have a method to fetch more items without overwriting the default "always load" (causing some pretty strange bugs).

Marco van Herwaarden
09-18-2005, 05:01 AM
Sorry Kirby, i have my hands full with RL work and family stuff at the moment. I will look at this as soon as i have some time.

Zachariah
03-30-2006, 12:32 AM
Bump - I forgot about this, looking @ his again, just waken others up to an old subject.

[move to Coders area] ?

Code Monkey
01-11-2007, 04:37 PM
Anyone still messing with this?

I set this up for a private mod way back then on one of my sites. Now I can't install vBSEO on that site even though it runs fine on other sites on the same server. I am wondering if this is the culprit. It's the only thing different.

EDIT: I can confirm that this mod does interfere with vBSEO.

thincom2000
01-22-2007, 08:41 PM
Did this NOT get added when 3.6 came out?

patrickb
08-21-2007, 07:45 AM
Has there been any updates on this?

Marco van Herwaarden
08-21-2007, 08:07 AM
This has been addressed in later versions of vBulletin, so this discussion is outdated.

Closing thread for that reason.