CODERS PLEASE DISCUSS!!! Pre-Datastore modifications with XML-files (UPDATE 28-7)

[Marco van Herwaarden]

Coding in this thread should be considered a Proof of Concept, not to be used outside testboards!!!!!

Alternative 2 presented

Ok now i have got your attention, please read the following. I really want this to become a combined effort of the vbulletin.org community. If we can pull this of in a good way we could try to either convince Jelsoft that this should be standard (i doubt they will do this in 3.5, but worth a try), or we could use it as a standard extension that can be used by many coders. Or everybody will just shoot me for this idea.

We got all these nice plugins etc, but there is still something that can not be done without file edits:
Adding datastore items ($specialtemplates)

Before it also seemed impossible to touch prefetching templates ($globaltemplates & $actiontemplates) or new Phrasegroups ($phrasegroups), although i think most of these could be done with a plugin nowadays

The datastore seems untouchable because it gets loaded before the plugin system gets active.

Alternative 1

See file: Pre-Datastore using XML-files - v1.00.zip

I made 1 modification to ./includes/init.php that could solve this problem. What it does is give you the chance to create a XML-file for your hack that will be loaded before the datastore gets build.

To use this a 1 time modfication to init.php would be needed (unless Jelsoft wants to use it in the standard product), that could be shared by all hacks.

It let's you add in a controled way entries to the following vars:
- $phrasegroups
- $specialtemplates
- $globaltemplates
- $actiontemplates (not working in this version, would have to do some small changes for it)

I have been testing it on my localhost and it doesn't add much to the pageloads (important, it would be executed on each page load).

Please all coders have a look at this modification and give feedback (positive/negative or improvements)

In ./includes/init.php find:

PHP Code:

 $db->timer_start('Datastore Setup');
} 


Add after:

PHP Code:

// #############################################################################
// Start Hack: Pre-Datastore Plugin (MarcoH64)
// Optional TODO: Check $xml['product'] is active
// Not sure if this would be needed in this situation, it can't harm much if it is always active, unless a lot of removed plugins
// But in that case loading might get slow also
// Reading the product table for now to see if active, this extra query will slow down things, so maybe just remove
// We might also want to store in teh $vbulletin object, so it won't need to be queried later again
// Retrieve active products
$actproducts = $db->query_read("SELECT productid FROM " . TABLE_PREFIX . "product WHERE active = 1");
$actprods = array();
while ($actproduct = $db->fetch_array($actproducts))
{
$actprods[] = $actproduct['productid'];
}
unset($actproducts);
// / Optional TODO: Check $xml['product'] is active
require_once(DIR . '/includes/class_xml.php');
// Maybe seperated directories for the different types of xml-files could speed up processing of the 
// list of files. List might get long with a lot of hacks installed. Example:
// ./includes/xml/cpnav <- Only ACP navigation
// ./includes/xml/preds <- Only Pre-Datastore
$handle = opendir(DIR . '/includes/xml/');
while (($file = readdir($handle)) !== false)
{
if (!preg_match('#^preds_(.*).xml$#i', $file, $matches))
{
continue;
}
$preds_file = $matches[1];
$xmlobj = new XMLparser(false, DIR . "/includes/xml/$file");
$xml =& $xmlobj->parse();
// Optional TODO: Check $xml['product'] is active
// Always allow product 'vbulletin' :))))))))))))
if ($xml['product'] != 'vbulletin'
AND !in_array($xml['product'], $actprods))
{
continue;
}
// / Optional TODO: Check $xml['product'] is active
 
if (!is_array($xml['predsgroup'][0]))
{
$xml['predsgroup'] = array($xml['predsgroup']);
}
 
foreach ($xml['predsgroup'] AS $predsgroup)
{
// Check if we should handle this entry for the current script
// If the entry is marked with script=global, then always include
// If no script is given, this entry is poorly written and should be distrusted and skipped
// If vb_area is not set in xml-file, or no match, skip. This is done to protect from running while in ACP if lazy coders ommit the area
if ((trim($predsgroup['script']) != 'global' AND trim($predsgroup['script']) != THIS_SCRIPT)
    OR (trim($predsgroup['vb_area']) != VB_AREA))
{
continue;
}
 
// Handle adding to $specialtemplates
if ($predsgroup['specialtemplates'])
{
if (!is_array($predsgroup['specialtemplates'][0]))
{
    $predsgroup['specialtemplates'] = array($predsgroup['specialtemplates']);
}
$specialtemplates = (is_array($specialtemplates) ? array_merge($specialtemplates, $predsgroup['specialtemplates'][0]) : array_merge(array($specialtemplates), $predsgroup['specialtemplates'][0]));
}
 
// Handle adding to $phrasegroups
if ($predsgroup['phrasegroups'])
{
if (!is_array($predsgroup['phrasegroups'][0]))
{
    $predsgroup['phrasegroups'] = array($predsgroup['phrasegroups']);
}
$phrasegroups = (is_array($phrasegroups) ? array_merge($phrasegroups, $predsgroup['phrasegroups'][0]) : array_merge(array($phrasegroups), $predsgroup['phrasegroups'][0]));
}
 
// Handle adding to $globaltemplates 
if ($predsgroup['globaltemplates'])
{
if (!is_array($predsgroup['globaltemplates'][0]))
{
    $predsgroup['globaltemplates'] = array($predsgroup['globaltemplates']);
}
$globaltemplates = (is_array($globaltemplates ) ? array_merge($globaltemplates , $predsgroup['globaltemplates'][0]) : array_merge(array($globaltemplates ), $predsgroup['globaltemplates'][0]));
}
 
// Handle adding to $actiontemplates
if ($predsgroup['actiontemplates'])
{
if (!is_array($predsgroup['actiontemplates'][0]))
{
    $predsgroup['actiontemplates'] = array($predsgroup['actiontemplates']);
}
$actiontemplates = (is_array($actiontemplates) ? array_merge($actiontemplates, $predsgroup['actiontemplates'][0]) : array_merge(array($actiontemplates), $predsgroup['actiontemplates'][0]));
}
}
$xmlobj = null;
unset($xml);
}
// End Hack: Pre-Datastore Plugin (MarcoH64)
// ############################################################################# 


Save and upload init.php.

That is all.

How to use this?

As said before you can now use a XML-file uploaded to the ./includes/xml' directory to control things.

XML-File naming: ./includes/xml/preds_<productid>.xml
Replace the <productid> preferable with the productid you assigned to your hack, but any other unique name should also work.

Now the content of the file:

<?xml version="1.0" encoding="ISO-8859-1"?>
<predsgroups product="productid">
<predsgroup script="global" vb_area="AdminCP">
<phrasegroups>phrasegroup</phrasegroups>
<specialtemplates>datastoreitem</specialtemplates>
<globaltemplates>globaltemplate</globaltemplates>
<actiontemplates>actiontemplate</actiontemplates>
</predsgroup>
<predsgroup script="showthread" vb_area="Forum">
<phrasegroups>phrasegroup</phrasegroups>
<specialtemplates>datastoreitem</specialtemplates>
<globaltemplates>globaltemplate</globaltemplates>
<actiontemplates>actiontemplate</actiontemplates>
</predsgroup>
</predsgroups>

Note: Like said before actiontemplates don't work yet, so don't even try to use them.

- You must set a product! (if your product is not enabled, code will not run from this file.
- script must contain the name of the file for which this group should be loaded, or 'global' to always load.
- vb_area MUST be set. Default vB has 1 areas 'Forum' and 'AdminCP'
- You can have as many predsgroup's as you need.
- Within each group you can have as many lines as you need. Only the above tags are supported, all other will be ignored.


Please all have a look at this 'proof of concept' and give feedback.

If we think this would be a nice way of standarization with only once a code edit (unless.....) then i will release this to use for all hacks released here on vb.org, if the coder wants to use it.


Alternative 2

See for details file: Proof of Concept Pre-Datastore for plugins - Alternative 2 Version 1.00.zip

Description:
Proof of Concept Pre-Datastore for plugins - Alternative 2 Version 1.00
This code is part of a proof of concept discussion at vbulletin.org. Goal of this discussion is to create a way
to add new datastore items in hacks without having to modify source code for each hack.
Unless Jelsoft decide to incorporate one of the presented alternatives this goal will never be met. At least there would be needed
a one time code modification for all hacks:
Thread: https://vborg.vbsupport.ru/showthread.php?t=93007
Author: MarcoH64
This code might NOT be used as a whole or a part without authors permission.
*** THIS CODE IS STRICTLY TO BE USED FOR CONCEPTUAL TESTING, AND NOT TO BE USED OUTSIDE A TESTBOARD ***
================================================== ================================================== =
Alternative 2:
Add a new table that will store information of datastore items needed for any script.
Merge the info of that table with the standard vB datastore items that are retrieved in unmodified code.
The idea of this alternative was presented by different members on vb.org and vb.com.
Pro in this solution:
- No need to parse XML-files for each pageload (compaired to Alternative 1)
- Should probably be faster because of this.
- Table can be loaded together with the default Product XML-file
Con's in this alternative:
- Will also execute if Plugin System is globally disabled (chicken and egg problem, or extra queries needed)
- A lot more file modifications needed
- Script to maintain the information stored in the table not written (yet?).
Should anyway only be available in debug mode?
- Need to create a new table
- Query to use when loading the datastore on each page load might be slowed down considerably. This should be tested
in a representable environment. Same goes for alternative 1.
- This alternative only handles new datastore items, not template caching like alternative 1

[deathemperor]

exactly what I was thinking of how I was supposed to deal with uncached templates and things listed above.

A very nice hack, Macro. I hope vb.com will consider this.

[Marco van Herwaarden]

Let's make it perfect as a combined effort, and maybe we can also as a combined effort try to get it into standard vB.

[Andreas]

Hmm, it adds 1 Query to every Page being loaded, and parsing several XML Files for every Page also adds Overhead.
Global and Action Templates can already be added through Plugins, not sure if this is possible for Phrasegroups too - but I think Phrasegroups might not be an issue at all, as those moste likely would be used in additional Files anyway.

So what's left is Datastore.

I posted a somewhat similar idea @ vbulletin.com some time ago:
http://www.vbulletin.com/forum/showthread.php?t=146141

IMHO, this should not add any Overhead at all.

Or, to keep your idea:
As it requires an additional Query, what about the following:

A Table requirements with colums

• scriptid
• globaltemplates
• actiontemplates
• phrasetypes
• specialtemplates
• product
• area

init.php would then query this table for the current scriptid/area, joined on products that are active.
Then merge the results with the current values and continue execution.

[Adrian Schneider]

Thanks Macro, I will test this out today, as I was having this problem earlier.

[Marco van Herwaarden]

Originally Posted by KirbyDE

Hmm, it adds 1 Query to every Page being loaded,

Yes that is also why i wrote in my comment that we should discuss if testing if a product is active would be something that is really needed.

Keep in mind please that i also want this to be a disccusion, and if we all agree that this test is not worth the extra query, we could remove it easy.

Originally Posted by KirbyDE

As it requires an additional Query, what about the following:

A Table requirements with colums
scriptid
globaltemplates
actiontemplates
phrasetypes
specialtemplates
product
area

init.php would then query this table for the current scriptid/area, joined on products that are active.
Then merge the results with the current values and continue execution.

That would be an even more heavy query i guess.

Another alternative to the solution i have choosen here would be to load all this data from the XML-files (should be loaded on install of a product then) into a table, and then add this data serialized (hmm now i remember i wanted to add support for serialized datastore items, will do this tomorrow) to the datastore. This new record in the datastore shoudl always be queried when loading datastore items, and the load routine for the datastore could then dynamicly add those items to the datastore. But this would also mean another query (first query to retrieve this new record, then another to retrieve additional information, or retrieve all records from datastore, but only store the wanted ones into the $vbuleltin->datastore)

[Marco van Herwaarden]

Originally Posted by TheSpecialist

Thanks Macro, I will test this out today, as I was having this problem earlier

I ran into the datastore problem myself when working on the vB.org Hack Database. And i just don't want to be editing 10 files, only to add the datastore item and do all other things with plugins.

[amykhar]

I haven't been able to get the phrasegroup working in a plugin and I need it to work for two existing files. I ended up just making the phrases global phrases, but that's not optimal.

Amy

[Andreas]

Originally Posted by MarcoH64

That would be an even more heavy query i guess.

Yes. But as the Tables won't be big it should not make much of a difference; at least it should be faster then opening and parsing a bunch of Files.

Hmm, it might be even possible to achieve smth. without additional Queries:

Have all requirements (eg. Phrases, Global and Action Templates - except Datastore Items) as a serialized DS Item that is always being loaded.
Combined with the Idea in the linked Thread above, it should not require any further Queries.

or retrieve all records from datastore, but only store the wanted ones into the $vbuleltin->datastore)

Wouldn't that require way more memory/database bandwidth?

[Marco van Herwaarden]

You mean you want to load ALL phrases and templates into the datastore???

Edit: PS Maybe you should update your signature