vBulletin 3 Hacks & Quality Assurance at vB.org

[KuraFire]

Stefan, perhaps a better idea would be to make vB3 hacks appear in the same column as the vB2 hacks, but underneath or above it, so that people can easily distinct between them without us having to add [vB3] or [vB2] to each thread title.

[Bison]

I got here a bit to late to review all of the threads written here and I don't have the time to do this but I'll throw in my two cents on this subject:

There is a definite need to create a rating system for hacks at this forum. In the past, there have been many modified versions of the same hack ... some just a bit more prettier than it's predecessor. Most of them riddled unnecessary queries. This is what I believe ... created too many help tickets at vb.com, and most of the problems were related with what we have been producing here.

Think that there should be two separate categories ... if a hacker doesn't submit his hacks to the rating team, his hacks should be allowed, but put in a category that tells members that this hack was not reviewed by the rating team. This will leave it up to the user to decide if they want to install this hack. The other category lists the hacks that have been reviewed and graded for functionality ... and rated accordingly.

If the hacker doesn't like thier score, they have the option of requesting that their hack be removed from this list so that they can make the necessary changes to bump up their rating. They have to apply the reccommended changes that came from their previous review in order for the hack the be re-released to the reviewed forum.

Am I making sense?

[LightBringer]

Great idea, but a question does come to mind.
Resources required to review the hacks.

With as quickly as hacks are released around here, do you guys have all the peeps necessary to keep the ebb of flow going to the community so A: We're not waiting around for months for an in-progress hack, and B: Support for those hacks are still given in a timely manner?

[KuraFire]

Originally Posted by Bison

I got here a bit to late to review all of the threads written here and I don't have the time to do this but I'll throw in my two cents on this subject:

There is a definite need to create a rating system for hacks at this forum. In the past, there have been many modified versions of the same hack ... some just a bit more prettier than it's predecessor. Most of them riddled unnecessary queries. This is what I believe ... created too many help tickets at vb.com, and most of the problems were related with what we have been producing here.

Think that there should be two separate categories ... if a hacker doesn't submit his hacks to the rating team, his hacks should be allowed, but put in a category that tells members that this hack was not reviewed by the rating team. This will leave it up to the user to decide if they want to install this hack. The other category lists the hacks that have been reviewed and graded for functionality ... and rated accordingly.

If the hacker doesn't like thier score, they have the option of requesting that their hack be removed from this list so that they can make the necessary changes to bump up their rating. They have to apply the reccommended changes that came from their previous review in order for the hack the be re-released to the reviewed forum.

Am I making sense?

Yeah you are, but whenever a Hack is not approved, the Hack author can fix the flaws in his/her Hack and just re-submit the new version for approval again, already.

So what you suggested was already part of the plan. \o/
(which makes it a good suggestion )


I don't think we want to make too large a distinction between approved and unapproved hacks, though. Too much segregation ;D is not productive for people's spirits.

[KuraFire]

Originally Posted by LightBringer

Great idea, but a question does come to mind.
Resources required to review the hacks.

With as quickly as hacks are released around here, do you guys have all the peeps necessary to keep the ebb of flow going to the community so A: We're not waiting around for months for an in-progress hack, and B: Support for those hacks are still given in a timely manner?

We're working on that

Keep in mind that even Hacks that are in the queue will be visible on the list and can be used and have support etc.

[Princeton]

For usabiltiy purposes...
I believe that all new hacks should be placed in a BETA HACK forum. If the percentage of 'yea' votes are higher then the 'nea' votes (80/20) then an automatic move to the HACK forum is enabled. (cron job)

If a hack is found in the HACK forum, a member shouldn't be obligated to look for approval. It should be assumed that it has been approved.

The BETA HACK forum should exist for purposes of bringing out quality hacks and quality coders.

[alkatraz]

having 2 levels or whatever is a great idea!

[KuraFire]

Originally Posted by princeton

For usabiltiy purposes...
I believe that all new hacks should be placed in a BETA HACK forum. If the percentage of 'yea' votes are higher then the 'nea' votes (80/20) then an automatic move to the HACK forum is enabled. (cron job)

If a hack is found in the HACK forum, a member shouldn't be obligated to look for approval. It should be assumed that it has been approved.

The BETA HACK forum should exist for purposes of bringing out quality hacks and quality coders.

Not a very viable option because that would mean that we are to either review every single hack that is released, and thus it's sort of an enforcement (ie. hack authors have no choice anymore about whether they want their hack reviewed or not). That, or we are to not review any hacks at all, which defeats the whole purpose of this plan.

Letting normal members approve of a hack with votes is not a reliable way of defining whether the Hack is safe to use or not. Best example would be Lesane's Store Hack. That would easily get a ton of votes, but that doesn't make it a safe-to-use Hack.

We're putting this together to provide a 'safety label', a 'quality label' if you will. Hacks that are QA-Approved are thusly reliable that you can be sure of it that if you install the hack, someone will not be able to break into your admin panel through some security leak that the Hack created.

[Princeton]

Here's what I think should be done:

• NEW RELEASES should be APPROVED HACKS.
• All hacks are sumbitted into BETA HACKS forum for review/approval.
• Judges and members review hacks to make end-product optimized and approved. Upon approval hacks are "moved" to APPROVED HACKS.
• Judge's comments are highlighted (using vbcode so that coders/members) can distinguish who is a judge.

Not a very viable option because that would mean that we are to either review every single hack that is released, and thus it's sort of an enforcement (ie. hack authors have no choice anymore about whether they want their hack reviewed or not). That, or we are to not review any hacks at all, which defeats the whole purpose of this plan.

1. The "rating" system does not necessarily have to be open to all members. I never said it should. On the other hand, the BETA HACKS forum SHOULD be open to ALL members (which it is).
2. By entering BETA HACKS forum, a member already assumes that hack may not be up to par. If a hack is submitted in BETA HACKS forum any member can use such hack at their own discretion. At the same time, "judges" will not have a backlog of reviews to implement at any one time. Isn't the BETA HACKS forum available to coders for such a review? Will judges install and test all hacks?
3. This system will make any hacks immediately available to everyone. BECAUSE, at any one time you cannot gaurantee that all "judges" will be available. As the saying goes, anthing can happen.
4. Furthermore, any hacks not approved may discourage authors to submit additional hacks or requests a review. At the very least, submitting such a hack into the BETA HACKS forum will allow authors to improve such hack using recommendations from everyone especially the "judges".
5. Upon approval, not the coder; but, "judges" may move such hacks to APPROVED HACKS.

As ADMINs:

• You should think about the workload.
• You should think about usability.
• You should think about how people behave/react.
• All criticism is good for coders. A critique made for one coder can and will help other coders. However, a guideline should be created on "how" to critique. We do not want people to degrade others or feel degraded.
• There should be a defualt beta hacks questionaire for judges; and, answers should be visible on hack page (showthread) with comments on how to improve.
• I believe, also, there should be a usability factor. Not to determine "approval"; but, to make aware to all users who are not familiar with what is usable. How usable is this hack?
• There should be a "probationary period" for members who are found to go against these rules. To let a member know that they are in probation, admins should change style to reflect a "probationary period". For example, a red highlighted border on page; and, for usabiltiy purposes the written words "You are in probation ... read more here" with a heading style. It will be used as a reminder to what could come after --> BAN.

Here's a scenario:
Upon the release of VB3 RC1 an influx of new hacks will be submitted for approval. Why? Because, all coders want to release a clean and optimized hack. Will your reviews be available instantly? Or, will coders have to wait-in-line to get such approval? And, will such hack be available online without a seal-of-approval?

[Catch-22|BL]

Peer review? Please just make sure you have some good reviewers!

Let me depart from the existing discussion and just reply to Erwin's request for comments.

I propose six hacking levels (tongue in cheek):

Rust
Iron
Bronze
Silver
Gold
Platinum


I propose this flowchart (half-kidding):

1. Someone submits a hack.

2. Important: When a hack first submits his/her code, they should identify if they are dumping the code (as is) or if they are sticking around. Is it okay for other people (with credit provided) to modify or "adopt" their hacks if they vanish?

3. A qualified person does a clean install to see if the thing even works. This person should also be familiar enough with vb.org to spot duplicates and blatant rip-offs.

4. If it does not delivered promised function, the hack is put in the Rust (disfunctional hacks) category.

5. If it works, the qualified person estimates the relevance and it is either put in the Iron (appears to function but is really just a trivial tweak) or Bronze (appears to function and has some significance). Note the Iron category is good enough for most custom requests that will only be used by a handful of people. No further reviews are necessary for Iron level and that is as far as it goes for those hacks. Insignificant hacks do not merit the time or energy. If by some miracle an Iron hack starts getting dozens of installs, someone can always bump it up to Bronze.

6. Bronze hacks can either stay at that level or the hacker can request an upgrade to Silver status. If the hacker requests the status upgrade, some very qualified people review his/her code and observes what happens when the first few people start installing the hack. A couple weeks in the shark tank will let you know if a new hack is poorly optimized or messy. If the hack is found to be fully functional and sufficiently optimized, the hack is sent to the Silver level. If not, the hack stays Bronze and the hacker is told to either abandon his/her request or improve the hack.

7. After a month at Silver, the hacker can request an upgrade to Gold. Unless flaws have appeared, it should be simply a check to see if the hacker is supporting the hack and addressing comments from the community. Aftercare is the measuring stick for Gold.

8. Once at the Gold level, it should be automatic that the hack works on the current version of software and the zip file should be a self-contained unit. Combing through support threads that are hundreds of posts long should definitely not be necessary. Things should by tidy.

9. Any Gold level hacks that reach a certain number of installs can be automatically promoted to Platinum (highest level). Platinums hacks should be functional, significant, sufficiently optimized, well-maintained and popular.

10. Important: Hacks need to be reviewed and perhaps demoted on a regular basis. If a hacker stops supporting his/her Gold hack, it should be dropped back down to Silver. If there are so many upgrades that a hack loses function or relevance, it should be moved to a lower priority.




Before you laugh too hard, think about it....basically any person who stumbles across a decent idea and is willing to humbly revise his hack (sometimes under the guidance of others) while remaining persistant can proudly proclaim they have a Gold level hack. Quality over quantity, baby.