Subscriptions is the name of what you need... included as a default feature in vBulletin for 2 years already... in a lot of colors.

no need of hack. you give the ability to your members to pay a membership, which will give them permission to a specific forum, or even a feature, like access to private messaging, editing others messages, etc.

and vBulletin can be used with Paypal, and 5 other default payement systems.. and you can even ask to have other ways to pay.