phraseid numbers ..... big problem! ... i warned that this would happen!

[trafix]

Hi all ...

I raised concerns a week or so ago in a pvt forum with regard to the possibility of 2 hacks using the same phraseid no. i suggested that a phraiseid number registration system should be put in place so that this cant happen,

I proposed before a hack (that requires its own new phrase id) is released the author must register the phraseid with vb.org so it dosnt clash with another.

Most of the replies to my concerns said that it would be unlikley to happen ..... WELL IT HAS!

My members choice hack has clashed with the geek articles hack ..... and is causing the installers of both hacks major headaches as you would be able to understand ... just read page 4 of the members choice voting hack thread.

Before you start accusing me that i used a phrase id for the members choice that i know would clash ... think again .... i released the members choice for vb3 origionally on vb.nl when vb3 hacks couldnt be released here. I used the same phraseid on the release here as on vb.nl

Lets get this phrasesid registration system coded on vb.org, before it happens again .... if it already hasnt with other hacks. It should be very easy to code a system here to automate the registration on phraseid's

trafix

[Marco van Herwaarden]

Yes it is very likely that this would happen one day. I think the same goes for the permission bitfields.

[Paul M]

Am I missing something obvious here ? - Why not just use 'null' in the insert code when you install the phrase, and let mysql allocate the number ?

[trafix]

unfortunatly that wont work :(

When setting up a new phrase group you need to add a row to the phraseid db table and then the phrases added to the phrase db table have to echo the same id number .... null wont work unfortunatly

[Paul M]

Sorry, are you talking about the phraseid or phrasetypeid ?

[trafix]

oops both

you need to add the row to the phrasetypeid table and then phrases to the phrase table

i spose you could add it the phrasetypeid with null .... but then you would have to quiry what phrase id was allocated then use a string to add the phrases ...

But what about hack updates .... its gonna need a more complex installer again to assertain the phraseid before any new phrases are added .........

then there is another problem that a hack uses x phraseid for your forum and y if its installed on my forum ......

I think its better if the phraseid stays unique for every installation of the hack

[Xenon]

shouldn't the phrasetype name be unique for each hack?
so if you know the typename, then you can get the type id, and with that id, you can get the correct phrases.

actually there i don't really see a problem.

the bitfields on the other hand could be a bit more problematical, as these would need users who know the details behind them before installing a hack...

[trafix]

shouldn't the phrasetype name be unique for each hack?
so if you know the typename, then you can get the type id, and with that id, you can get the correct phrases.

actually there i don't really see a problem.

the bitfields on the other hand could be a bit more problematical, as these would need users who know the details behind them before installing a hack...
Yes the phrasetype name and phrasetypeid have to be the same in both phrasetype and phrase db tables.

as it is at the moment when coders add a new phrasegroup for their hack they pluck a number out of the sky and use it ....... unfortunatly the coder of the geekarticles and myself plucked out the same number to use as the phraseid .....

if you read the members choice hack thread ... you would see the conflict that has been reported and basicly the best way to fix the problem is that both hacks have to be un installed and reinstalled using different phrase id nos

PS thanks Xenon for you fast reply to this matter

trafix

[Paul M]

Ah, I see what you are talking about now - the phrase types.

The phrasetypeid field in the phrasetype table is not an auto-increment field, so you cannot add a new record with a null value and expect mysql to allocate a value to it - you have to supply a number.

However, the phraseid, in the phrase table is auto-increment, and you can add actual phrases with a null value and let mysql number them - this is what I thought you were referring to as you quoted phraseid rather than the typeid.

Also the typeid is an unsigned smallint - which means you only have up to 65,500 (ish) to play with.

I would suggest using something like typeid = (10000 + int(time() / 10000) to give you a unique id at installation time (the value of this would currently be around 21,000 and changes every 2.75 hours).

[trafix]

Ah, I see what you are talking about now - the phrase types.

The phrasetypeid field in the phrasetype table is not an auto-increment field, so you cannot add a new record with a null value and expect mysql to allocate a value to it - you have to supply a number.

However, the phraseid, in the phrase table is auto-increment, and you can add actual phrases with a null value and let mysql number them - this is what I thought you were referring to as you quoted phraseid rather than the typeid.

Also the typeid is an unsigned smallint - which means you only have up to 65,500 (ish) to play with.

I would suggest using something like typeid = (10000 + int(time() / 10000) to give you a unique id at installation time (the value of this would currently be around 21,000 and changes every 2.75 hours).
as far as i am aware ..... it is coded into vb to ignore phraseids above 1000 ... whick narrows it down from 65,000 odd :(

[Paul M]

as far as i am aware ..... it is coded into vb to ignore phraseids above 1000 ... whick narrows it down from 65,000 odd :(Is it ? I wonder why ......

Oh well, the installation script could just run a query to find the largest current typeid, and add 1 ......

[Floris]

Is it ? I wonder why ......

Oh well, the installation script could just run a query to find the largest current typeid, and add 1 ......
I wouldn't know .. but I think its simply a limitation of field storage in mysql ?

[Xenon]

hmm, i wonder why phrasetypid doesn't have the autoincrement.

Did the devs just forgot it?
or has it a reason, would be nice to know

[Paul M]

I wouldn't know .. but I think its simply a limitation of field storage in mysql ?As I said above, the field [in the mysql table] is an unsigned smallint, so it can range from 0 to 65535, trafix seems to imply the limitation has been coded into vB for some reason. I haven't checked it, just taken his word as correct. :)

[trafix]

well actually im not sure how it works but spacial phrase groups use id nos above 100 for ex
email body = 2000
email subject = 3000
something else uses 9000

but they are all multiples of 1000

and what about all the phrase ids that are currently scattered between where vb stops and 1000 i have used
153 for the members choice ... i will have to change it
888 for the jukebox .... i dont know if anyone else is using it
152 for the trader .... i dont know if anyone else is using it

there again if i change the idno for the members choice ... i have to code the installer to check which one is used before updates can be made

[trafix]

hmm, i wonder why phrasetypid doesn't have the autoincrement.

Did the devs just forgot it?
or has it a reason, would be nice to knowi think its delibrate as the phraseid have to be identical in both tables

[Marco van Herwaarden]

Well the phrasetypeid is a real problem with the way hacks are made right now, but this could be solved with some intelligent installer. The question is however, do we want that for 1 hack different phrasetypeid's will be used on every installation. That don't make life easier.

About the permission bitfields, i really think this is a much bigger problem. These normal require manual file edit's, where you must manually check in the sources if a wanted number is still free, and if not decide on a new number. With deciding on a new number it is very easy for a non-coder who don't have god knowledge about how bitfields work, to choose the "wrong" number.

I think it would be a great improvement to hacks released for vB (and thus an improvement of the experience users have with vB) if coders can make good agreements on this, and that jelsoft will support this with a registration system of used id's.

[Revan]

oops both

you need to add the row to the phrasetypeid table and then phrases to the phrase table

i spose you could add it the phrasetypeid with null .... but then you would have to quiry what phrase id was allocated then use a string to add the phrases ......Either that, or we could use $DB_site->insert_id(); ;)

[Dean C]

Just make a function to check if the phraseid/phrasegroupid is available. Then if it's not try another one. It's not hard :) This was done with vB2 hacks and the setting/settingroup table :)

[Marco van Herwaarden]

As I said in the other thread too, just code it into your installers to check what settingid's are available and then just insert them with that ID. It's not rocket science and used to be done on many installers in vB2 :)
And how to handle the permission bitfields then? Not in this way. Maybe we can find a way if we make good agreements among hack developers.

[Dean C]

Permission bitfields is a different issue. There's no real way I can see at the moment of getting around it other than documenting what to do when there is a clash :)

[Marco van Herwaarden]

Hmm must think, there must be something possible i think, but that would involve some piece of standard code (buildin? SDK?) that all coders should include in their hacks if they wanted to use bitfield permissions.

[trafix]

Just make a function to check if the phraseid/phrasegroupid is available. Then if it's not try another one. It's not hard :) This was done with vB2 hacks and the setting/settingroup table :)
Yes but what ever is decided ALL coders have to follow the same path .... if they dont the whold thing will screw up again.

how damn easy is it just to register phraseid nos or bitfields into a db table for each hack that requires it .... it would be very easy to code and even easier for coders to comply ..... if the number that they want to use is taken by another hack, think of another number ..... then mycates and others like him wont have to manually uninstall 2 hacks and start over

alright, erm well i unistalled this hack manually and i've removed the geekarticles manually. I will try to install it again later today. I'll let you know how i get on.



Thanks


what could be more simple?

[trafix]

PS .... it minght make plp installing hacks ask if the phraseid has been registered ... or even look it up in the db ...... i would think twice about installing a hack if i knew that the install could crash due to another hack using the same phraseids no's .....

Why are we mastercating about this?

[Revan]

For bitfields, I think the system is abit flawed.
I would think it was a better idea to have a function parse through the array of bitfields (so instead of "[fieldname] => [number],", it would just be "[fieldname],"), adding the incremental bitfield just like we have to do manually now, before running the function currently being ran to prepare the bitfield for SQL insertion.
Then, when the bitfield SQL is converted to array again, a sub-piece of the above function or a different function would parse through the array of bitfields one more time, this time matching up the two arrays thus assigning the correct bitfield with its correct permission field.
This would cause no extra queries, and very little PHP parsing time (unless a bitfield array had 10,000 array keys in it...).

But what do I know, $5 says the vB devs have a much better way lined up for the next vB :)

[trafix]

How many vb noobs out there would have read this thread and know now that there have been a crash between 2 hacks are wanting to install any hacks now incase it happens to them? i wouldnt be feeling very secure installing any hack to my board atm.....

This issue needs urgent addressing before any more crap happens .... its just as well mycates is relativly experiances with php and db's

[Marco van Herwaarden]

I'll second that trafix.

Would also be nice if coders together could think of a way that more general functions fon't need to hack into the same place of php-files for every hack over and over again.

For a non-coder (or even for coders ;) an error in editing a file is easilly made, so the less edits, the better.

[Andreas]

Maybe I am dumb, but I don't see the problem (except the limited range for phrasetypeids maybe).

If you call add_phrase_type() it will allocate a free id?

[Dean C]

Yes but what ever is decided ALL coders have to follow the same path .... if they dont the whold thing will screw up again.

Nope, just include this method in your modification and your users won't have a problem. What other modification authors decide to do is up to them :)

[trafix]

im not fremilliar with "add_phrase_type()" as a function, i will have to investigate it :)

Maybe I am dumb, but I don't see the problem (except the limited range for phrasetypeids maybe).

If you call add_phrase_type() it will allocate a free id?

[Marco van Herwaarden]

About the phrasetype's, this means that all coders will now change their installers to support automatic finding the next available id?
Can't really say that i am satisfied with the response What other modification authors decide to do is up to them
I would be giving support if i wrote an installer according to the "rules" and later some other hack don't assign them dynamic and overwrite my phrasetypes, causing my hack to fail.


And what we can do with the bitfields? Something we can agree on together to avoid future problems?

Remember that in the end the users (admins) of the hack will blame the "product vBulletin" if things don't work how they want or cause them problems. Wether this can be blamed on teh core product or not.

[Natch]

Finding the next available phrasetype ID is the most responsible way to do this: you cannot be responsible for toher coders, only yourself.

Making a mountain out of a molehill is not the way to get around a molehill.

[Andreas]

@MarcoH64

About the phrasetype's, this means that all coders will now change their installers to support automatic finding the next available id?

If all coders would use the standard vBulletin function add_phrase_type() instead of using their own code there would have never been a problem.
So I think nobody can blame vBulletin or Jelsoft for causing problems, only the coders who don't use common API.

As for bitfields: I'd vote to make a note in the manual that standard bitfields should be left untouched.

[Carnage]

adding extra bitfields is easy isn't it? its always seemed that way to me... just multiply the last number in it by 2 not hard to write some dynamic code modifications to deal with that...

let me throw some code together... and work this out.


<?php

include("includes/init.php");

foreach ($_BITFIELD AS $KEY => $VALUE) {
echo "<b>" . $KEY . "</b><br>";
foreach ($VALUE AS $L1KEY => $L1VALUE) {
if (!is_array($L1VALUE)) {
echo "--" .$L1KEY . "=>" . $L1VALUE ."<br>";
} else {
echo "<b>--" . $L1KEY . "</b><br>";
foreach ($L1VALUE AS $L2KEY => $L2VALUE) {
echo "----" .$L2KEY . "=>" . $L2VALUE ."<br>";
}
}

}
}
?>


the above makes the bitfield echo out quite nicely so you can see whats allready there. step 2 is to create something to dynamically create the code to add and where to add it. be back in 20mins or so :p


<?php

include("includes/init.php");

$addto = "adminpermissions";
$addwhat = "candoanything";

//create new bit field
foreach ($_BITFIELD AS $KEY => $VALUE) {
if($KEY != $addto) {
foreach ($VALUE AS $L1KEY => $L1VALUE) {
if($L1KEY != $addto) {
continue;
} else {
echo "<pre>\$_BITFIELD['" . $KEY . "']['" . $L1KEY . "'] = array(<br>";
foreach ($L1VALUE AS $L2KEY => $L2VALUE) {
echo " '" . $L2KEY . "' => " . $L2VALUE . ",<br>";
$lastvalue = $L2VALUE;
}
$newvalue = $lastvalue * 2;
echo " '" . $addwhat . "' => " . $newvalue . "<br>);";
}
}
} else {
echo "<pre>\$_BITFIELD['" . $KEY . "'] = array(<br>";
foreach ($VALUE AS $L1KEY => $L1VALUE) {
echo " '" . $L1KEY . "' => " . $L1VALUE . ",<br>";
$lastvalue = $L1VALUE;
}
$newvalue = $lastvalue * 2;
echo " '" . $addwhat . "' => " . $newvalue . "<br>);";
}
}

?>


not 100% complete but you get the idea... its not hard to do.

[Paul M]

@MarcoH64

If all coders would use the standard vBulletin function add_phrase_type() instead of using their own code there would have never been a problem.
So I think nobody can blame vBulletin or Jelsoft for causing problems, only the coders who don't use common API.

As for bitfields: I'd vote to make a note in the manual that standard bitfields should be left untouched.Where is this (and all the other) "common" API's documented ?

[Andreas]

includes folder :)

@carnage
Well, even if you assign the value dynamically you can never be sure that future vBulletin versions won't ust it for its own purposes - and then ppl will wonder why hacks that rely on a certain bit don't work any longer.

[Carnage]

well if they do when the instal script is re-run to install it after the upgrade, the bit field will be dynamically assigned to a higher number.

[Marco van Herwaarden]

well if they do when the instal script is re-run to install it after the upgrade, the bit field will be dynamically assigned to a higher number.
Yes but you would loose all permissions set prior to upgrade in best case scenario. Worse case the 'old' permissions of that hack are now being used (without being reset, since they in the eyes of jelsoft already where all 0) by some new standard permission, giving unwanted people access.

[Andreas]

Worst-Case Scenario:
Jelsoft introduces a new Bit "Is Superadmin" (eg. does have all admin privileges) in usergroup permissions - while your hack used it for <place smth. harmless here>.
Now users unaware of this upgrade and - BANG

[Paul M]

includes folder :)Huh ?

They are not documented anywhere as far as I can see, people are not going to use things they don't know about.

[Natch]

I think he was saying that if you want to know what a function does, the best documentation is to look aat the function, and work out what it does.

If this is not something you can do, then your next point of reference is php.net and mysql.com ... that's teh way the rest of us learned how-to.

[Carnage]

Worst-Case Scenario:
Jelsoft introduces a new Bit "Is Superadmin" (eg. does have all admin privileges) in usergroup permissions - while your hack used it for <place smth. harmless here>.
Now users unaware of this upgrade and - BANG

i tried that... adding is superadmin requires far too meny code changes to be worth the extra conviniace of being about to superadmin people from the cp.

but yeah i see this point; i suppose this is why hacks are not officially supported... probably the only solution is to warn users of the problems it can (and will?) cause and to make sure that they check permissions etc after installing/upgrading.

or alternativly... use your own bitfield... however myself i quite like the admin permissions bitfield and have added half a dozen new permissions to it for my warnings system and there will probably be more to come...

[Andreas]

Not supporting hacked boards is one thing, "intentionally" increasing the danger of security leaks is another.

As said, this is a hypothetical worst-case scenario and most likely won't ever happen.
But it is far more dangerous to use an unused bit for custom things then adding a column to a table (the upgrade script would crash with a mySQL-error if a column with this name does already exist when they try to ALTER TABLE) or introducing new variables (this can of course also cause problems, but only if Jelsoft uses the same variable name; Bitfields will be used sooner or later)

Btw: Creating such a "issuperadmin" usergroup permission IMHO would be pretty easy: In init.php check the bit, if it is set also set ismoderator and cancontrolpanel.
In can_administer() check that bit and if it is set return true, no matter what $do is.

[bluesteel]

How many vb noobs out there would have read this thread and know now that there have been a crash between 2 hacks are wanting to install any hacks now incase it happens to them? i wouldnt be feeling very secure installing any hack to my board atm.....

This issue needs urgent addressing before any more crap happens .... its just as well mycates is relativly experiances with php and db's

He he he trafix.
Spent feckin loads of attempts over the past weeks trying to install your Jukebox hack. Fails every time with a MySQL error. After spending another hour messing about with it tonight, plus coming across this thread, realised that Jukebox is trying to use phraseid 888, which is already in use by the Quiz hack by TWCommish that I have installed.
Bugger! :(

[trafix]

He he he trafix.
Spent feckin loads of attempts over the past weeks trying to install your Jukebox hack. Fails every time with a MySQL error. After spending another hour messing about with it tonight, plus coming across this thread, realised that Jukebox is trying to use phraseid 888, which is already in use by the Quiz hack by TWCommish that I have installed.
Bugger! :(
Oh crap! :(

Tnx for the feedback bluesteel ...... just goes to show the importance of phraseid number registration .....

you can change the number in the quiry.php and the phrase.php .... where is says 888 change it to another unused phraseid, and it should install :)

[bluesteel]

No worries mate. Sorta figured that out.
Now I'm off back to the JB hack thread to figure out other small probs :)

[Carnage]

Not supporting hacked boards is one thing, "intentionally" increasing the danger of security leaks is another.

As said, this is a hypothetical worst-case scenario and most likely won't ever happen.
But it is far more dangerous to use an unused bit for custom things then adding a column to a table (the upgrade script would crash with a mySQL-error if a column with this name does already exist when they try to ALTER TABLE) or introducing new variables (this can of course also cause problems, but only if Jelsoft uses the same variable name; Bitfields will be used sooner or later)

Btw: Creating such a "issuperadmin" usergroup permission IMHO would be pretty easy: In init.php check the bit, if it is set also set ismoderator and cancontrolpanel.
In can_administer() check that bit and if it is set return true, no matter what $do is.

Yes well, i'm not going to release the hack in question since i didn't write most of it in the first place... so it'll only be on the board it was written for. In the event of any upgrades, I can easilly change admin permissions if the need arises.

the superadmin user permission is slightly more difficult than that, you have to go through and find every check for the In_array($superadmins_array) thing(can't remember the exact script) and replace it with a check to that permission, however as a quick way to grant all normal permissions, yes it is quite simple.

My intention with it was to have it function in the same way we had root admins function on vb2, root admins were the ones who controlled admin permissions, they were the only ones able to grant access to the admin and root admin forums, plus they had a bunch of other useful things such as the ability to change peoples userid and to add elements to the rpg system we used. However it turned out alot more difficult than i though it would be I may have another go at doing something like this when i've got other things sorted out.

[trafix]

Is something happening with a registration system ??????

I think this is now urgent!!!!

So har now 2 of the 4 major hacks i have released clash phraseid's with another hack.

the members choice clashed with the geek articles hack

and

my jukebox is clashing with some quiz hack.

I WANT YOU TO TELL ME WHAT IS THE POINT OF ME RELEASING ANY MORE HACKS IF I HAVE TO SPEND EVERY SECOND POST HELPING OTHER TO FIX THEIR FORUMS BECAUSE OF CLASHING

[Andreas]

Is something happening with a registration system ??????
There is no registration system necessary for phrasetypes ...

[trafix]

so what is your solution that stops clashes?

[cinq]

I've read through this thread (with slight understanding of what is going on).

But I have a query on the usergroup permissions bitfields.
Can't we already assign the own group name ?
eg.


$_BITFIELD[\'usergroup\'][\'myownpermissions\'] = array( ....


Would this still cause a clash ?
(other than some other hack using EXACTLY the same 'myownpermissions' naming.

[Andreas]

so what is your solution that stops clashes?
Use add_phrase_type() to create your custom phrasegroups :)

[trafix]

I've read through this thread (with slight understanding of what is going on).

But I have a query on the usergroup permissions bitfields.
Can't we already assign the own group name ?
eg.


$_BITFIELD[\'usergroup\'][\'myownpermissions\'] = array( ....


Would this still cause a clash ?
(other than some other hack using EXACTLY the same 'myownpermissions' naming.
yes Cinq ... i did that for the jukebox permissions :)

Use add_phrase_type() to create your custom phrasegroups :)
ok then it is very likley that every forum the hack is on will have a different phraseid no for the same hack ....

that is well and good but what about when you release upgrades to the hack .... there do the new phrases go?

[Andreas]

If it returns false (eg. not created), it most likely does already exist => check table phrasetype to get the ID (if you need it).

[trafix]

coding an installer for vb2 was childs play .... however vb3 is very complicated in the way it installs templates and phrases/phrasegroups etc.

I wouldnt know where to start ...

The HTL is dead .... and the HIS obviously dosnt cut the mustard.

How you you suggest that i code and installer that is smart enough to install a hack properly as well as being intelegent enough to find spacific numbers before and upgrade can be performed?

Can you code one for me?

[Andreas]

Hmm, I thought HIS does handle phrases already?
I'll take a look at it but might take some days.

[Cap'n Steve]

This is the same basic problem that I've encountered after starting a hack. Lack of documentation is a huge problem for software like this that is attractive largely because of the hacking community. The developers have promised a reference manual with the next major version, but until then we just have to read every php file (and wish there were more comments).

[trafix]

Hmm, I thought HIS does handle phrases already?
I'll take a look at it but might take some days.
Yes the HIS does do phrases ..... BUT .... there are several phrase groups that it fails with that I know of so far

Email Body phrases and control panel stop messages.

The HIS will not collect these files when auto generating the phrase.php file and if the phrases are edited manually into the phrase.php in installing a hack the email body phrases are not installed. I have mentioned this to Link but he dosnt believe me :(

[Paul M]

ok then it is very likley that every forum the hack is on will have a different phraseid no for the same hack ....

that is well and good but what about when you release upgrades to the hack .... there do the new phrases go?I can think of at least two ways to find the phraseid used (for upgrades).

1. Store it in the DB somewhere.

2. Run a query to find a phrase you know exists, and extract it's id.

[trafix]

Paul .... i am aware of the options, please feel free to build it into an installer for me ;)

[trafix]

has anyone got the majic solution yet????

Im waiting and i dont realy want to release any more hacks until this problem is fixed.

1) register phrase ids
2) intelligent installer that all coders must use to avoid clashes

[Andreas]

require_once('./includes/adminfunctions_language');
$phrasetypeid = add_phrase_type('myphrases', 'These are my phrases');
if (!$phrasetypeid) {
// Hmm ... smth. went wrong. Let's see if it already exists
$check = $DB_site->query_first("SELECT phrasetypeid FROM " . TABLE_PREFIX . "phrasetype WHERE fieldname = 'myphrases'");
if ($check['phrasetypeid']) {
// Oh, my phrasegroup does already exist!
// Guess this is an update then
// My phrasetypeid is $check['phrasetypeid'] and I can use it now the insert phrases
} else {
// W00t !?! My phrasegrouop can't be created AND does not exist yet?
// Fatal Error, call 911 and wreak havoc!
}
} else {
// Fine! My phrasegroup has been created, its ID is $phrasetypeid and I can use it now to insert phrases
}