Like i said you have a compromised file somewhere..
I tried all this:

Upload all new vbulletin files & clean up the database - no effect
Ok so then i thought i will htaccess the forum dir (password protect it) nope still loads of emails bouncing back.

Right i will htaccess the whole website, still nope still emails

Ok then lets close the website from inside admincp right? .. wrong still loads of emails.

All righty then i will remove all hacks and plungins even delete their files.. Nope! still emails

OK disable plugins in the config.php file.. nope nothing worked.

So then i started to think maybe it was my server so checked all the setting and found nothing wrong with the server, maybe a few brute force attacks but that was all..

By this time his email account on his website was being filled that much that i got server admin emails warning me that that user has sent umteen thousand and was reaching their limit.

So because it was for a friend and he was not bothered about the few posts on there he said i could delete and he would start a fresh.. WOW no more emails..

Hope it helps..

Unfortunately a re-install isn't an option.

I'm working my way through logs but off the top don't see anything related to sendmessage.php or other obvious vbulletin php.

I do see a fair number of errors that look like the following, but googling suggests are comment spam, not php mail spam.

forums/index.php+++++++++++++++++++++++++++++++++++++Resu lt:+\xed\xe5+\xed\xe0\xf8\xeb\xee\xf1\xfc+\xf4\xee \xf0\xec\xfb+\xe4\xeb\xff+\xee\xf2\xef\xf0\xe0\xe2 \xea\xe8;+Result:+\xed\xe5+\xed\xe0\xf8\xeb\xee\xf 1\xfc+\xf4\xee\xf0\xec\xfb+\xe4\xeb\xff+\xee\xf2\x ef\xf0\xe0\xe2\xea\xe8;, referer: http://URLRemovedByDoob.com/index.ph...0%E0%E2%EA%E8;

You could also try looking through your plugins to see if you notice any that you don't recognize.

I'm pretty sure its not a plugin problem.

I just ticked off "Allow Users to Email Other Members" under AdminCP->VbulletinUptions->Email Options.

I'll have to wait and see if that has any effect. Next step probably to disable Email Functions on that same page and switch to SMTP and see if that has any effect.

May also be forced, belatedly, to upgrade to current patch, however in googling around this seems to affect folks running versions well into the 4.1.x strata.

You may be right, I don't know. But I have seen plugins that somehow got added that allowed hackers to do anything from any page by including parameters (and posted parameters don't go in the logs so you wouldn't see it there).

Edit: I should add that I don't have a lot of experienced with hacked sites or anything, I've just seen a few posts about it on the forum.

I just ticked off "Enable Email features?" under Email Options as the next step in testing. This really isn't how I saw my day going.

Have you checked for suspect files under admincp>maintainance>diagnostics to make sure all your core files are correct?

Thanks for that sugg. I already checked. The only discrepencies are core files I edited myself.

So no files that you don't recognise then? check files outside of your forum root, you may have a file or two you don't recognise. Your server logs should show which file has been sending mail or accessed a hell of a lot.

Do you have "Use Mailqueue System" set to on? Not that that's a problem of course, but if you had lots of emails queued then the logs showing what happened could be a long way back, and also disabling the options might not immediately stop mail from going out (I don't know if turning off email features clears the queue, but there's no check for that when mail from the queue is being sent out). Also I don't know of there's a way in vb3 to see what's in the queue from the adminCP, but you could look at the mailqueue table directly.