If you want to use SMTP in vBulletin instead of using the local sendmail/mail function of your server, then I'm afraid it's not possible.
vBulletin makes use of fsockopen when SMTP is used, which has no option to disable certification verification.
Of course you can make modifications to /includes/class_mail.php and add something like this
http://i.justrealized.com/2009/allow...and-fsockopen/
https://stackoverflow.com/questions/...when-using-tls