Well, since you've been pretty much ignored, let's see if I can shed some light on this. First, let's make certain I understand what you're asking for.
You want to make your forum registration non-public, by invitation only, giving a special URL or password to the invited person.
This way, no one can register unless they have an invitation sent to them by a current member.
Is this correct?
If so, a hack which addresses part of your request can be found
here. It offers a special registration URL which automatically registers in the referral system, but does not preclude anyone else from registering.
Somewhere, someplace, I recall a hack which does pretty much all of what you want - I'll have to see if I can find it. Then again, I may be recalling an IPB or UBB hack. If I find it, and it is for vB, I'll return and let you know. Or someone else may have an answer.