I got this working on vbadvanced and vbulletin 3.7. Pretty sure these aren't the best directions to do get this work, but I'm just sharing what I did to get it working.
My setup:
vbadvanced:
www.yourdomain.com
vbulletin:
www.yourdomain.com/forums
Before installing, I edited the xml file. I added the URL to the forums where it says private.php. There are 2 places to edit.
If you already, installed the mod, I would uninstall and make the following changes.
Before:
HTML Code:
<phrase 1="private.php?$session[sessionurl_q]">$vbphrase[gtpm_new_private_message]</phrase>
After:
HTML Code:
<phrase 1="http://www.yourdomain.com/forums/private.php?$session[sessionurl_q]">$vbphrase[gtpm_new_private_message]</phrase>
----------
Before:
HTML Code:
<phrase 1="$newpm[username]" 2="$newpm[title]" 3="private.php?$session[sessionurl]do=showpm&pmid=$newpm[pmid]" 4="$stylevar[imgdir_statusicon]">$vbphrase[gtpm_have_new_pm_popup_x_y]</phrase>
After:
HTML Code:
<phrase 1="$newpm[username]" 2="$newpm[title]" 3="http://www.yourdomain.com/forums/private.php?$session[sessionurl]do=showpm&pmid=$newpm[pmid]" 4="$stylevar[imgdir_statusicon]">$vbphrase[gtpm_have_new_pm_popup_x_y]</phrase>
----------
1) Install the mod.
2) Go to: vbulletin admin control panel -> vBa CMPS -> Default Settings -> Main Options
In "Portal Output Global Variables", add: newpm
3) Click save.
That should be all.