The vBulletin 5 documentation is pretty poor so you'll probably have to figure most things out yourself. In the code the only reference to invoking a hook seems to be function invoke($hook_name, $params) which can be called using (an example taken from the new login hook):
PHP Code:
vB::getHooks()->invoke('hookProcessNewLogin', array(
'result' => &$result,
'logintype' => $logintype,
'cssprefs' => '',
'userinfo' => $newsession->fetch_userinfo(),
));
The available hooks and classes seems to be imported from the installed products which you can see at /core/vb/vb.php function getHooks. The invoke function can be found at /core/vb/utility/hook/live.php function invoke.
By default vBulletin 5 comes with a couple of packages/products that interact with these hooks so you can see what kind of code you can put inside of hooks and how the params are passed to the hooked function. One example can be found at /core/packages/googlelogin/hooks.php and the other files in that folder.
Hopefully that will help you out for now.