1- Very likely your problem is related to the mail server, not vbulletin. I'd suggest you contacting your hosting company and tell them "a PHP script can not send mail" and ask what is the problem.
2- If your board is not hacked, you can ask for official support in vbulletin.com too. I'm %99.99 sure this is not related to vbulletin but maybe you'll relieved better if you hear it from officials.