To be clear, when you set up SMTP in vBulletin, you use the same email access that you do for your regular pop email account on your ISP service or gmail. Make sure you have provided that information in the boxes in Email Options in vboptions, for example:
SMTP Email Yes
SMTP Host mail.yourdomain.com
SMTP Port 25
SMTP Username
you@yourdomain.com
SMTP Password youpassword
Run the email test in diagnostics and you should get a test mail in your domain inbox.
HTH