You need to recode some parts of your main site in order to work with vBulletin. According to your response I take it that your main site just utilized the same code as phpBB uses to check login authentication e.t.c.
You can obviously achieve the same with vBulletin but it will take some work converting your scripts over. Check out the Articles section here for information (especially
How to create your own vBulletin-powered page! (uses vB templates)) although unless your fluent in PHP you may want to consider hiring someone to do this work for you.