vB4 is faster. vB5 currently has some performance issues which are very apparent on large sites. Just test out vbulletin.com/forums these days (as a logged in user) and you will see it is slower now than when it was running vB4.
Most of the new software versions (PC & Web) are usally slower than the earlier versions. More features .... less speed.... is a natural consequence...
VB3 is better performing than vB4 which is better performing than vB5 for large websites. For small websites with just a few hundred visitors per day, it does not make any difference.
I wouldn't go so far as to say vB3 is faster. I think there are performance issues with it, over vB4. vB5 still seems to be working out it's speed issues.