Fringehead, you may be able to fix things by modifying the rewrite rules yourself to reflect your subdomains and other changes.
The rewrite code in gallery assumes you are running on a single domain. I also don't think you are going to be able to share cookies amongst both subdomains, so you are stuck with passing session id's in the url which I have found to be fairly slow.
As for the other problem with your gallery appearing inside a vbulletin window, thats easy to fix. Open the template for gallery2 and change it to:
Yep... thats it. This keeps all the vbulletin headers, css and other things far away from your embed gallery the way it should be =)
Good luck on mod_rewrite, it's a nice site you have there.