after you add the sub-domain forum.mydomain.com in your cpanel, all you need to do is add
http://forum.mydomain.com into your vbulletin options>> Site Name / URL / Contact Details>>Forum URL and refresh. i cant remember having to do anything else but that.
add Subdomains in cpanel to point to forum folder forum.mydomain.com.
then update your new Forum URL in admincp options.