CODERS PLEASE DISCUSS!!! Pre-Datastore modifications with XML-files (UPDATE 28-7)
 

Coding in this thread should be considered a Proof of Concept, not to be used outside testboards!!!!!

Alternative 2 presented

Ok now i have got your attention, please read the following. I really want this to become a combined effort of the vbulletin.org community. If we can pull this of in a good way we could try to either convince Jelsoft that this should be standard (i doubt they will do this in 3.5, but worth a try), or we could use it as a standard extension that can be used by many coders. Or everybody will just shoot me for this idea. :D

We got all these nice plugins etc, but there is still something that can not be done without file edits:
Adding datastore items ($specialtemplates)

Before it also seemed impossible to touch prefetching templates ($globaltemplates & $actiontemplates) or new Phrasegroups ($phrasegroups), although i think most of these could be done with a plugin nowadays

The datastore seems untouchable because it gets loaded before the plugin system gets active.

Alternative 1

See file: Pre-Datastore using XML-files - v1.00.zip

I made 1 modification to ./includes/init.php that could solve this problem. What it does is give you the chance to create a XML-file for your hack that will be loaded before the datastore gets build.

To use this a 1 time modfication to init.php would be needed (unless Jelsoft wants to use it in the standard product), that could be shared by all hacks.

It let's you add in a controled way entries to the following vars:
- $phrasegroups
- $specialtemplates
- $globaltemplates
- $actiontemplates (not working in this version, would have to do some small changes for it)

I have been testing it on my localhost and it doesn't add much to the pageloads (important, it would be executed on each page load).

Please all coders have a look at this modification and give feedback (positive/negative or improvements)

In ./includes/init.php find:
Add after:
Save and upload init.php.

That is all.

How to use this?

As said before you can now use a XML-file uploaded to the ./includes/xml' directory to control things.

XML-File naming: ./includes/xml/preds_<productid>.xml
Replace the <productid> preferable with the productid you assigned to your hack, but any other unique name should also work.

Now the content of the file:
Note: Like said before actiontemplates don't work yet, so don't even try to use them.

- You must set a product! (if your product is not enabled, code will not run from this file.
- script must contain the name of the file for which this group should be loaded, or 'global' to always load.
- vb_area MUST be set. Default vB has 1 areas 'Forum' and 'AdminCP'
- You can have as many predsgroup's as you need.
- Within each group you can have as many lines as you need. Only the above tags are supported, all other will be ignored.


Please all have a look at this 'proof of concept' and give feedback.

If we think this would be a nice way of standarization with only once a code edit (unless.....) then i will release this to use for all hacks released here on vb.org, if the coder wants to use it.


Alternative 2

See for details file: Proof of Concept Pre-Datastore for plugins - Alternative 2 Version 1.00.zip

Description:
Proof of Concept Pre-Datastore for plugins - Alternative 2 Version 1.00
This code is part of a proof of concept discussion at vbulletin.org. Goal of this discussion is to create a way
to add new datastore items in hacks without having to modify source code for each hack.
Unless Jelsoft decide to incorporate one of the presented alternatives this goal will never be met. At least there would be needed
a one time code modification for all hacks:
Thread: https://vborg.vbsupport.ru/showthread.php?t=93007
Author: MarcoH64
This code might NOT be used as a whole or a part without authors permission.
*** THIS CODE IS STRICTLY TO BE USED FOR CONCEPTUAL TESTING, AND NOT TO BE USED OUTSIDE A TESTBOARD ***
================================================== ================================================== =
Alternative 2:
Add a new table that will store information of datastore items needed for any script.
Merge the info of that table with the standard vB datastore items that are retrieved in unmodified code.
The idea of this alternative was presented by different members on vb.org and vb.com.
Pro in this solution:
- No need to parse XML-files for each pageload (compaired to Alternative 1)
- Should probably be faster because of this.
- Table can be loaded together with the default Product XML-file
Con's in this alternative:
- Will also execute if Plugin System is globally disabled (chicken and egg problem, or extra queries needed)
- A lot more file modifications needed
- Script to maintain the information stored in the table not written (yet?).
Should anyway only be available in debug mode?
- Need to create a new table
- Query to use when loading the datastore on each page load might be slowed down considerably. This should be tested
in a representable environment. Same goes for alternative 1.
- This alternative only handles new datastore items, not template caching like alternative 1

exactly what I was thinking of how I was supposed to deal with uncached templates and things listed above.

A very nice hack, Macro. I hope vb.com will consider this.

Let's make it perfect as a combined effort, and maybe we can also as a combined effort try to get it into standard vB.

Hmm, it adds 1 Query to every Page being loaded, and parsing several XML Files for every Page also adds Overhead.
Global and Action Templates can already be added through Plugins, not sure if this is possible for Phrasegroups too - but I think Phrasegroups might not be an issue at all, as those moste likely would be used in additional Files anyway.

So what's left is Datastore.

I posted a somewhat similar idea @ vbulletin.com some time ago:
http://www.vbulletin.com/forum/showthread.php?t=146141

IMHO, this should not add any Overhead at all.

Or, to keep your idea:
As it requires an additional Query, what about the following:

A Table requirements with colums

• scriptid
• globaltemplates
• actiontemplates
• phrasetypes
• specialtemplates
• product
• area

init.php would then query this table for the current scriptid/area, joined on products that are active.
Then merge the results with the current values and continue execution.

Thanks Macro, I will test this out today, as I was having this problem earlier. :)

Yes that is also why i wrote in my comment that we should discuss if testing if a product is active would be something that is really needed.

Keep in mind please that i also want this to be a disccusion, and if we all agree that this test is not worth the extra query, we could remove it easy.
That would be an even more heavy query i guess.

Another alternative to the solution i have choosen here would be to load all this data from the XML-files (should be loaded on install of a product then) into a table, and then add this data serialized (hmm now i remember i wanted to add support for serialized datastore items, will do this tomorrow) to the datastore. This new record in the datastore shoudl always be queried when loading datastore items, and the load routine for the datastore could then dynamicly add those items to the datastore. But this would also mean another query (first query to retrieve this new record, then another to retrieve additional information, or retrieve all records from datastore, but only store the wanted ones into the $vbuleltin->datastore)

I ran into the datastore problem myself when working on the vB.org Hack Database. And i just don't want to be editing 10 files, only to add the datastore item and do all other things with plugins.

I haven't been able to get the phrasegroup working in a plugin and I need it to work for two existing files. I ended up just making the phrases global phrases, but that's not optimal.

Amy

Yes. But as the Tables won't be big it should not make much of a difference; at least it should be faster then opening and parsing a bunch of Files.

Hmm, it might be even possible to achieve smth. without additional Queries:

Have all requirements (eg. Phrases, Global and Action Templates - except Datastore Items) as a serialized DS Item that is always being loaded.
Combined with the Idea in the linked Thread above, it should not require any further Queries.

Wouldn't that require way more memory/database bandwidth?

You mean you want to load ALL phrases and templates into the datastore???

Edit: PS Maybe you should update your signature :D

Not really, bandwidth is not really an issue, since it is just a server process, and the datastore table will never have hundreds of records.

Edit: Memory used to keep the datastore in memory for later processing by scripts is more an issue then the memory used while retrieving everything.

I will try to make some alternatives tomorrow.

What I meat was a serialized array with the Requirements for each Product.
This item should always be queried then, the Load Routine should extract the currently needed Data (identified by scriptid/area) and merge it with the existing $globaltemplates, $actiontemplates, $phrasegroups and continue execution - just like it does with the XML Files idea.

I will have a sleep over this all, not sure yet that what you are suggesting could be done with less performance issues then the solution i posted above, but maybe we should write out all possible solutions and find some big boards who want to be testing it and marking execution times. On my localost, the above solution (including the query) added 0.01 second to the page load.

PS I made a reply at your thread on vb.com, asking others to join us here in the discussion.

I am not sure either :)
But in Theory a Query (if it's not heavy) should be faster then File Operations and Parsing.

I say it should load all the entries from the XML files regaurdless of whether the product is enabled or not to save the query.

I say Kirby's original idea at vB.com sounds good with a few tweaks for datastore stuff.

I will also have a go at implementing that idea tomorrow, unless someone else volunteers. In the end i hope that we come together with the best solution and try to persuade Jelosoft in integrating it, or we could use it here for all coders who want to use it.

Great idea, here are some comments

1. We really should find something to eliminate the need for an extra query and an xml parse for each page view. Large sites cannot afford this. Of course smaller sites can use this, but hack authors would still require filed edits, so their hacks don't slow down large sites.

2. Since there are file edits anyway, why not edit global.php instead. I think some re-arrangement can help, such as moving init_language to below a hook (can't see any side effects, but I could be wrong). This, I assume will solve the phrasegroup issue?

3. What's left are templates. But since we can already modify global templates with a plugin, and if it uses an if statement to check for the script being used before merging the hack's template, then we're done (except for special templates)

But, let's hope vB Dev's surprise us before gold, and we don't have to mock things up :) I noticed Scott's reply to Kirbey's thread, so I am optimistic.

Thanks for the input tamarian, will have a good sleep over this and try to find some alternatives tomorrow.

Didn't notice the reply at vb.com yet, but will go have a look.

To all other coders........input please.

It sounds like a great addition, but I would like to see what the Dev's take on this :)

Satan

I thought if you had another table all that would be required is to modify the datastore query that is already present to join another table with more conditions for extra things to pull

ie

[sql]
SELECT *
FROM datastore
JOIN NEW TABLE
WHERE title IN(blah) OR
NEW TABLE CONDITIONS (IF title=title and newtable.active)[/sql]

Second Alternative presented.

This one uses a table that gets merged during loading of the datastore rows.
Data is released inside the regular Product XML file that you use for your hack.

See newly attached file for details.

Again, please comment!!!!

If the plugin system is disabled, it could set a flag in the system (somehow) to also not load the extra datastore items, though it probably wont work because vboptions is in the datastore.

Not having looked at your proof of concept, I would have thought the only file modification needed would be in the datastore class to modify the query. It shouldnt be necessary on each pageload to do anything else..

I dont see creating another table as an issue - the more tables you have to deal with things the better! :) - I also dont believe that a query joined to this table would be slower than reading an XML file from the filesystem.

Template caching can already be achieved with a hook..?

Sorry if this goes off topic, but can you share your way to code this ?

I don't think that would chang a lot, since it would have to be stored somehow, it would also have to be retrieved, resulting in an extra query. (Not that an extra well designed query should always be such a big issue).
To implement it, only 1 modification to the datastore class is made. Ofcourse this added code must be executed on each pageload, how you would want to retrieve the info otherwise?

The other modifications are management code (loading the table with the product XML-File, disabling the entry when the product is disabled, etc.)
Creating another table is not an issue if Jelsoft is doing it, it is something to consider when a Third Party (us) is making changes to the database design.
I am taking the word of others on this, not tried it myself, but this was the beauty of Alternative 1 in my eyes. We could ofcourse always create something similar to the datastore hack for this. (Could be stored in datastore, so no extra query would be needed.)

Hook cache_templates

Uhh, yes - that is a Problem. Hmm, thinking about that.

Haven't really implemented it yet, but in theory it should be just a few lines in init.php?

Idea: Seriaized DS Entry that holds Phrase/Template Requirements for every Script. This entry would have to be loaded always, the currently required Data extracted and merged with $phraseggroups etc.

Already answered by:
Since our primary focus was on datastore and templates/phrases can already be done by plugins, this is not coded into Alternative 2. In alternative 1, i was able to easily catch all 4 arrays, within the same structure and edit location.

To implement phrases/templates into Alt. 2, completly different code would be needed compaired with the datastore solution (because the problem is different - adding things prior to having the datastore available - versus - adding things after we have the datastore). What they do have in common is that we could use the same table to store the info, with some slight modifications. I even started with columns in the table to support this, but i removed them later since i didn't plan to implement it.

PS We are being watched by Jelsoft on the progress in this thread (i think :D) and there might be a very slight chance that Jelsoft might adopt an idea (which would be the best solution for the community i think).

I am a bit disappointed at the lack of feedback and discussion in this thread from most coders.

Probably because having read it, I'm somewhat confused as to what the original problem is/was and what exactly you are trying to achieve. Perhaps I'll have another read after my holiday.

(and unrelated - but long thread titles with tons of CAPITALS in them are ones I tend to avoid ....)

So let me see if I understood this correctly - you want to be able to add new phrasegroups/actiontemplates to existing vB pages?

My suggestion:

AdminCP:
New file: Import/Export/Editing/Deletion of an XML file record containing list of templates/phrases required for each mod. A dupe of the plugin system, so to speak, only it contains a possiblity to amend to every single instance of the $xxxtemplates arrays, including subarrays.
Every time something is added/edited/removed from this list, a flatfile (similar to the flatfile DS cache option in vB) is created. Obviously it does array searches to make sure no duplicates are present (although they wouldn't do much harm).

Init.php:
Find
Add below
From there, different parts of init.php can use variables from this file (again, similar to the datastore_cache.php) to do whatever it is you want it to do, such as array_merge($precache_templates, $specialtemplates); etc.


A simple require_once() won't bring any huge load onto the server, it won't be parsing anything and it won't be querying anything.

The problem is then you have to CHMOD to install plugins.

I still say the new table/query modification is the best for datastore, and hooks can take care or everything else.

So what if you have to CHMOD 1 file once?
You don't have to CHMOD the current datastore_cache every time you rebuild the vBOptions if you use the flatfile method, do you?
And even IF you would have to, it's still more efficient than adding queries or doing advanced parsing.

That's what I've done in the past on my site. It does work. But would defeat the purpose of the concept of plugins.

Nice to see there is some life in this topic again.

Let me just try to get the original problem clear again:
- You can do a lot without code changes with 3.5, there is however 1 thing you can not: If your hack adds a new entry to the datastore, you will have to edit each file to add your datastore item to the list of items to be fetched. You can not use hooks, because they are only loaded after the datastore is read.
- If we can easily without extra effort also add a simple way to do the same for template caching, then do so. It would reduce the number of plugins needed for your hack.

The original idea was to come with a solution that Jelsoft even might consider implementing, but since we are already on RC2, i doubt that will happen. Second best would be a standarized way of handling this as a community, so that whatever hack you use, file edits would only have to be made once.

How does it defeat the purpose of the concept of plugins?
If the hack is extensive enough to require datastore plugging into, it will also require file upload, so CHMODding a file wouldn't be hassle.
I don't really know, so DO you have to re-CHMOD the datastore_cache.php each time you update vBOptions?
If you don't, I don't see any reason why this should be any different.

If I had known this would concern me I would have replied earlier, XD
For my RPG I require a cache, but I just created my own table (rpg_cache, datastore table duplicate) and query this whenever I call $RPG->init(). I never really wanted to plug into the datastore table, because I wanted my variables to be $RPG->item not $vbulletin->item without too much hassle :p
Im not saying this to say "that's the solution", fyi.

The easiest solution would just be to hassle Jelsoft into adding a hook or two to enable us to do what we are trying to do.
If we have to come up with an uniform way of doing this, however, I believe my suggestion's downsides are far smaller than the disadvantages of adding an extra query to pages who won't even need this new field, and parsing the XML file on the fly (Imagine what would happen if the user has 15 hacks that require datastore modification, *shudders*).

This is why i opened the discussion, to find the best together. Your way of adding your own table for the rpg hack works, but will add another query also. If many hacks do this, this will mean many queries. And using custom datastore items in your own new script is not a problem. It is using them in plugins into jelsoft scripts that require code edits.

@Revan
Not every Hack that does require a DS item necessarily has to come with it's own Files.
Furthermore, it's impossible to intruduce Hooks to handle the Datastore Problem, as Plugins itself are being loaded from the Datastore.
My idea does not introduce a new Query, it just changes an existing one to have a JOIN and 1 more Condition.
As the Tables that would be used are pretty small, this should hardly cause any Overhead.

The idea of keeping a var_dump()ed array in a File is also nice, as it does not cause parsing or Query Overhead.
vBulletin itself does so, if you use the Class for a File-Based Datastore, so I think the CHMOD issue isn't a big one, as this would have to be done only once.
Hmm, Safe Mode and open_basedir restrictions could cause Problems here though.

How can I make a custom class recognise custom DS items then?
Oh XD
How so? After all, this file could be stored in the same way as the current ds cache file, which obviously work with Safe Mode and open_basedir...

Easy require the special templates.

Eh? Gonna need abit more detail than that, girl :p