vBulletin 3 Hacks & Quality Assurance at vB.org
 

With the RC1 release of vBulletin 3 coming closer, the Staff at vBulletin.org wishes to raise the standard of code in Hacks released here. vB3 Hacks will soon be allowed here, and it is of great importance to us that the quality of Hacks goes up a notch. In the past (with vB2-hacks), plenty of hacks have been released here on vBulletin.org containing security loopholes, unnecessary queries, and overall bad code.
To raise the standard of Hacks, we are thinking of implementing a system where experienced, skilled and well-respected vB Hackers are selected to review vB3 Hacks released here at vB.org. This will come in a two-level flavor - a basic review that will be made for every single hack that is released, and a far more in-depth review (upon request).
Especially the second (and far more thorough) level of the system will allow end-users to easily spot the more reliable Hacks on the site, as the second level approval means it is a secure, well-coded, well-optimized, easy to install and overall good quality hack.

This system will hopefully encourage hackers to raise the standards of their coding. Obviously this is voluntary - no one is forced to follow the recommendations of this review team. The basic review that is applied on all Hacks is mostly to make sure that beta-quality hacks don't get posted as full releases, and other such things.

The rating criteria are not set in stone already, we're making this announcement so that we know what YOU all are interested in and find important etc.
We already have some ideas in mind. However, we wish to have input from the members too. In particular, we want to get suggestions for the following:

a) Criteria used to rate
b) What points sytem to use
c) What rating scale to use


Give us as many of your thoughts as possible on this, and we, the Staff, will put our heads together. The more you people can come up with, the better. Who knows, we may just end up with some really nice system that everyone just loves! It all depends on how well you all can think up useful ideas.

We hope that this will improve our community and the quality of hacks released, which will be good for everyone in the end. :)


Keep in mind that this will be site-wide for vBulletin 3 Hacks only, not vB2-Hacks. This does not mean that you can no longer release vB2 hacks, this only means that the rating system will not apply to them.

I think that's a great idea, Erwin.:)

By doing this, hacks will become even better and with a good team of hackers reviewing them, they will tie in is as well as possible with the standard code. I fully support it.:D

Excellent idea. Having people rate the standard of hacks will greatly improve the quality of future hacks:)

wonderful idea Erwin..... :D

I feel it's a good idea.
I plan on jumping on the VB3 hacks release bandwagon myself. As a upcoming PHP Coder.

Come on guys, we want your input on this, not just praise for the idea! Thoughts, Ideas, Comments, Criticism!!! Let's hear it! Anything at all?

Oh, and it wasn't only Erwin's idea, y'know :p

Sounds good, I've been getting my manual out, so I can make my code look nicer ;)

I fully support it. I think a 1 to 10 scale is just fine. I think it would be best if it was right down to the wire in the rating system. Like 7.6 Rating not just 8. Im sure you know what i mean, im a bit tired myself. lol

There should also be an area where not so great coders can post hacks too as well. Of course with this in mind, making sure all users know that this isnt the best coded hack.

Just kind of a thought I think would make it easier for peeps who aren't experts in PHP (I'm getting there, but by no means ready to code my own). With the downloads for each hack, maybe have some sort of rating/feedback system, one where you could on a scale of 1 to 10 rate various components of the hack, and to go with that, also have a comments section for each rating, like if you give installation a 7.8, explain why it wasn't a 10 (or why it was a 7.8 for that matter).

That might help identify more problem areas within a certain hack. I've seen a few that are easy to install, but required a lot of self coding to get going and then only to find out down the road it would generate some problems(*cough* Karma 2.0 *cough*).

just my 2 cents though

Good idea, although with a rating system, be prepared to see it abused, people hate people (that's human nature) and they will have fights with the rating system, inevitable.

Other than that, sounds good, and partly encouraging to the lesser experienced PHP programmers who will see this as an opportunity not only to have their work evaluated but to also advance and hone their PHP skills.

The ratings aren't going to be made public, so that should ease some of the verbal abuse in competitions.

noone will stop users from releasing hacks, even if they are bad coded.
it should just be visible to users that a hack is problematical/easy/good coded/well optimized or something like.

That's ok then.

As for the rating system, I think a % would be suitable, if not, scale it down to x / 10.

Again, ratings themselves will not be made public, not to the hackmaker nor to the general public of vB.org.

A Hack will either be approved or not approved. The Hackmaker will hear which aspects of Hacking/coding could be improved (whether the hack is approved or not), but (s)he will never hear the exact rating / score.

Why not? People will always get into arguments when they see a score. There will always be people who disagree with a certain rating, then there will always be others going around saying "my hack is better than yours because I got an 8.1 and you got a 7.5!", then you will have people trying to appeal to a decision and get a second opinion and what not. When you only release YES/NO approved to the public, there is very little to make a fuss about. And that's what we want - we don't want to cause a ruckus for each hack, we want things organized, clear, and useful. A big heated discussion isn't useful, nor well-organized. A set of guidelines on where you can improve on, is.

That said, we will also provide an extensive 'document' with help, tips and guidelines on how to make hacks, along with Kier who is writing such a thing already (last I heard).

As stated in the announcement, we want to improve the overall quality of all hacks released on vB.org, and we would most like to be doing that without causing a lot of fights/arguments among the crowds. :)

I'm a pretty basic php & MySQL coder - not up to hacking standards, but ok for installing them.

I'd REALLY appreciate this system, since I like adding hacks to my board, but need to keep it stable & secure.

Great idea!

Thanks Stefan, exacly my point.
People can release hacks galore if they want to compete in "some" competition... but if we have an input from people with experience, we can decide if we should wait or go for the mod.

I will even go one step further. Should we make the author being not able to see the comments? Not sure if it's a good idea...
This is with 2 sides... If you don't see the comments of people, you cannot get upset... ;)
However, how do you learn then from the coding mistakes? I'm sure you guys will find the right balance for it.... Let me know please.

Yeah, stopping the author view the comments makes no sense because it then defeats the purpose of the system:confused:

Its my understanding that hackers will be shown where they have gone wrong and pointed in the right direction so they learn from their mistakes and can improve :)

TECK, please PM me. Did you get my email? You have PM disabled so I cannot PM you.

A rating system would need to be adjustable as future incarnations of a hack arrive. If a bug-ridden 1.0 is replaced by a stable 1.1 (or whatever) then it may not deserve the potentially shoddy rating it received at the onset.

Feedback should be appreciated by any hack author (I know I appreciate any that I receive) but I am not sure that public critique is such a good idea (beyond the rating). If it were up to me, I would leave that private for the hack author. Having somebody put a lot of work into something only to see negative comments from experienced coders in public is not going to encourage them to keep working on it. Especially because something that takes me 20 minutes might take someone else 4 hours, or might take Chen 53 seconds, and so on. The chance of people thinking that other, more experienced hackers are "talking down" to them is high.

I think a publicly viewable stability rating (or whatever you want to call it) would suffice, along with private comments to the author about why such a rating is deserved and what he/she can do to improve upon it.

Keep the suggestions coming. :) We want the members to have input in to this system as it will affect the whole community.

Excellent idea indeed :)

The rating system doesn't have to be adjustable really, because we will give out this 'award' for only one (main) version of a Hack at a time. If the Hack is buggy at 1.0, it will simply not receive the Quality award. If the Hack at 1.1 is rewritten to have no more bugs (or no apparent ones, at the very least), the hack author can re-submit it for approval and we will then review it again. If at this point the Hack does indeed no longer have bugs, and is secure (no security leaks or dangers), and is running stable (doesn't randomly hog the server resources, or 'crashes'), it will be taken through a rating system for several criteria, and if it then scores well enough on those as well, it will be given the Quality label. If later on, then, a version 1.2 of the hack is released, it will have to be reviewed all over again, before we'll give the 1.2 version the Quality label.

After reviewing of a Hack, the hack author will be told (regardless of whether or not the Hack was approved) what aspects of coding s?he could still improve upon.
We will aim as much as possible to not be condescending in that, so that people don't get the feeling we're talking down to them or anything. We want to make this system to encourage people to improve their coding skills, and we know that acting all stuck-up and what not is not going to help with that, so we'll make sure to be supportive, friendly, and down-to-earth. :)
It's good that you pointed it out though. :):up:

The Public will only see one thing: QA Approved: yes/no; (no can also mean it simply has not been submitted for approval).
The Hack Author will see two things: QA Approved (duh), and what areas s?he could still improve upon. This will be referenced using an extensive document will lots of useful tips and guidelines.
The Quality Assurance team will see whether it's been approved or not, will see what areas the hack author could improve upon, and see in a little more detail how the rating process has gone. The latter is important to have for when we encounter hacks where one Team member thinks is -just- good enough to Qualify, and another Team member thinks it's comes a little short.


At least, that's planned so far... if any member sees a problem with that, please tell! It's why we made this thread - to get everyone's opinion on it :)

Kura: I guess what you are saying is a bit different than what others had said earlier in the thread. Having a simple Yes/No Approved seems like a better idea to me than a percent-based rating. Either it meets the standards or it doesn't.

As far as public comments, I still agree that the Quality Assurance team should communicate with the author in private.

This whole system sounds like a nice idea for the end user, but risks making it more complicated and less friendly to the hacker. Part of what I like about vb.org is that there are tons of different ideas implemented in tons of different ways here, and anybody is free to do it. I wouldn't want to see this system "crowding out" some of the guys who come in just to write one hack here or there.

The other risk is that you are making the process more complicated. Just like my opinion of the hack subforums: the more unnecessarily complex things become, the less useful they become.

This is a good idea, it just needs careful implementation.

I agree with the QA team discussing any issue with the author in private. People take comments the wrong the way at times. Maybe a standardized form should be filled out by the QA team and sent to author with basic comments and if the author has questions or simply disagrees with what the QA team "thinks" is the correct way to script something, the issue would be better handled in private. The less public arguements between hack authors and a QA team the better, imo for this site and people( end users) in general.

Discussions between Hack Author and QA Team concerning the submitted / approved / unapproved Hack will always be in private.

The rating criteria Erwin mentioned in the first post are for QA team in-house, the results of a review of a hack will not be made public via those. This to prevent endless bickering about details that aren't as important as the big picture: approved, yes or no?

grrrrr - eat as tony would say :D good idea chaps :D

I think this is a great ideal.

Also if it can use laymen terms as far as descriptions wrt install. I muttled through a chat hack, got it to work but it took a while.

Great ideal and vision, kudos

I love the idea Kura suggested about a step by step guide to creating a hack beeing written up! Im sure it will be VERY usefull to newbie/wannabe hackers like me.

Request--

When vB3 hacks are released can we have them shown on the front page like the vB2 ones are? I was thinking of a column next to the vB2 one, but i guess it might look out of place.

the vb3 hacks will be visible on the homepage yes.
most likely within the same window, we'll see

Stefan, perhaps a better idea would be to make vB3 hacks appear in the same column as the vB2 hacks, but underneath or above it, so that people can easily distinct between them without us having to add [vB3] or [vB2] to each thread title. :)

I got here a bit to late to review all of the threads written here and I don't have the time to do this but I'll throw in my two cents on this subject:

There is a definite need to create a rating system for hacks at this forum. In the past, there have been many modified versions of the same hack ... some just a bit more prettier than it's predecessor. Most of them riddled unnecessary queries. This is what I believe ... created too many help tickets at vb.com, and most of the problems were related with what we have been producing here.

Think that there should be two separate categories ... if a hacker doesn't submit his hacks to the rating team, his hacks should be allowed, but put in a category that tells members that this hack was not reviewed by the rating team. This will leave it up to the user to decide if they want to install this hack. The other category lists the hacks that have been reviewed and graded for functionality ... and rated accordingly.

If the hacker doesn't like thier score, they have the option of requesting that their hack be removed from this list so that they can make the necessary changes to bump up their rating. They have to apply the reccommended changes that came from their previous review in order for the hack the be re-released to the reviewed forum.

Am I making sense?

Great idea, but a question does come to mind.
Resources required to review the hacks.

With as quickly as hacks are released around here, do you guys have all the peeps necessary to keep the ebb of flow going to the community so A: We're not waiting around for months for an in-progress hack, and B: Support for those hacks are still given in a timely manner?

Yeah you are, but whenever a Hack is not approved, the Hack author can fix the flaws in his/her Hack and just re-submit the new version for approval again, already. :)

So what you suggested was already part of the plan. \o/
(which makes it a good suggestion ;))


I don't think we want to make too large a distinction between approved and unapproved hacks, though. Too much segregation ;D is not productive for people's spirits. :)

We're working on that :)

Keep in mind that even Hacks that are in the queue will be visible on the list and can be used and have support etc. :)

For usabiltiy purposes...
I believe that all new hacks should be placed in a BETA HACK forum. If the percentage of 'yea' votes are higher then the 'nea' votes (80/20) then an automatic move to the HACK forum is enabled. (cron job)

If a hack is found in the HACK forum, a member shouldn't be obligated to look for approval. It should be assumed that it has been approved.

The BETA HACK forum should exist for purposes of bringing out quality hacks and quality coders.

having 2 levels or whatever is a great idea!

Not a very viable option because that would mean that we are to either review every single hack that is released, and thus it's sort of an enforcement (ie. hack authors have no choice anymore about whether they want their hack reviewed or not). That, or we are to not review any hacks at all, which defeats the whole purpose of this plan.

Letting normal members approve of a hack with votes is not a reliable way of defining whether the Hack is safe to use or not. Best example would be Lesane's Store Hack. That would easily get a ton of votes, but that doesn't make it a safe-to-use Hack.

We're putting this together to provide a 'safety label', a 'quality label' if you will. Hacks that are QA-Approved are thusly reliable that you can be sure of it that if you install the hack, someone will not be able to break into your admin panel through some security leak that the Hack created.

Here's what I think should be done:

• NEW RELEASES should be APPROVED HACKS.
• All hacks are sumbitted into BETA HACKS forum for review/approval.
• Judges and members review hacks to make end-product optimized and approved. Upon approval hacks are "moved" to APPROVED HACKS.
• Judge's comments are highlighted (using vbcode so that coders/members) can distinguish who is a judge.

1. The "rating" system does not necessarily have to be open to all members. I never said it should. On the other hand, the BETA HACKS forum SHOULD be open to ALL members (which it is).
2. By entering BETA HACKS forum, a member already assumes that hack may not be up to par. If a hack is submitted in BETA HACKS forum any member can use such hack at their own discretion. At the same time, "judges" will not have a backlog of reviews to implement at any one time. Isn't the BETA HACKS forum available to coders for such a review? Will judges install and test all hacks?
3. This system will make any hacks immediately available to everyone. BECAUSE, at any one time you cannot gaurantee that all "judges" will be available. As the saying goes, anthing can happen.
4. Furthermore, any hacks not approved may discourage authors to submit additional hacks or requests a review. At the very least, submitting such a hack into the BETA HACKS forum will allow authors to improve such hack using recommendations from everyone especially the "judges".
5. Upon approval, not the coder; but, "judges" may move such hacks to APPROVED HACKS.

As ADMINs:

• You should think about the workload.
• You should think about usability.
• You should think about how people behave/react.
• All criticism is good for coders. A critique made for one coder can and will help other coders. However, a guideline should be created on "how" to critique. We do not want people to degrade others or feel degraded.
• There should be a defualt beta hacks questionaire for judges; and, answers should be visible on hack page (showthread) with comments on how to improve.
• I believe, also, there should be a usability factor. Not to determine "approval"; but, to make aware to all users who are not familiar with what is usable. How usable is this hack?
• There should be a "probationary period" for members who are found to go against these rules. To let a member know that they are in probation, admins should change style to reflect a "probationary period". For example, a red highlighted border on page; and, for usabiltiy purposes the written words "You are in probation ... read more here" with a heading style. It will be used as a reminder to what could come after --> BAN.

Here's a scenario:
Upon the release of VB3 RC1 an influx of new hacks will be submitted for approval. Why? Because, all coders want to release a clean and optimized hack. Will your reviews be available instantly? Or, will coders have to wait-in-line to get such approval? And, will such hack be available online without a seal-of-approval?

Peer review? Please just make sure you have some good reviewers! ;)

Let me depart from the existing discussion and just reply to Erwin's request for comments.

I propose six hacking levels (tongue in cheek):

Rust
Iron
Bronze
Silver
Gold
Platinum


I propose this flowchart (half-kidding):

1. Someone submits a hack.

2. Important: When a hack first submits his/her code, they should identify if they are dumping the code (as is) or if they are sticking around. Is it okay for other people (with credit provided) to modify or "adopt" their hacks if they vanish?

3. A qualified person does a clean install to see if the thing even works. This person should also be familiar enough with vb.org to spot duplicates and blatant rip-offs.

4. If it does not delivered promised function, the hack is put in the Rust (disfunctional hacks) category.

5. If it works, the qualified person estimates the relevance and it is either put in the Iron (appears to function but is really just a trivial tweak) or Bronze (appears to function and has some significance). Note the Iron category is good enough for most custom requests that will only be used by a handful of people. No further reviews are necessary for Iron level and that is as far as it goes for those hacks. Insignificant hacks do not merit the time or energy. If by some miracle an Iron hack starts getting dozens of installs, someone can always bump it up to Bronze.

6. Bronze hacks can either stay at that level or the hacker can request an upgrade to Silver status. If the hacker requests the status upgrade, some very qualified people review his/her code and observes what happens when the first few people start installing the hack. A couple weeks in the shark tank will let you know if a new hack is poorly optimized or messy. If the hack is found to be fully functional and sufficiently optimized, the hack is sent to the Silver level. If not, the hack stays Bronze and the hacker is told to either abandon his/her request or improve the hack.

7. After a month at Silver, the hacker can request an upgrade to Gold. Unless flaws have appeared, it should be simply a check to see if the hacker is supporting the hack and addressing comments from the community. Aftercare is the measuring stick for Gold.

8. Once at the Gold level, it should be automatic that the hack works on the current version of software and the zip file should be a self-contained unit. Combing through support threads that are hundreds of posts long should definitely not be necessary. Things should by tidy.

9. Any Gold level hacks that reach a certain number of installs can be automatically promoted to Platinum (highest level). Platinums hacks should be functional, significant, sufficiently optimized, well-maintained and popular.

10. Important: Hacks need to be reviewed and perhaps demoted on a regular basis. If a hacker stops supporting his/her Gold hack, it should be dropped back down to Silver. If there are so many upgrades that a hack loses function or relevance, it should be moved to a lower priority.




Before you laugh too hard, think about it....basically any person who stumbles across a decent idea and is willing to humbly revise his hack (sometimes under the guidance of others) while remaining persistant can proudly proclaim they have a Gold level hack. Quality over quantity, baby. ;)