[setishock]

20, 40 , 50, 200 380, 1000 milliseconds? That's thousandths of 1 second. That's incredibly fast for any software/hardware package. You have to take in to consideration your server box setup.
FSB > The bottle neck of all computer systems. If it's a slow buss then that will slow down data transfer.
Hard drive > Larger HD's have an inherantly longer seek time. More platters, more data, longer seek times. Bust a large drive into smaller drives and dedicate each one to a specific job.
Memory > Too fast combined with a slow FSB can cause congestion on the buss.
Check your northbridge specs. If it's too slow for the rest of the hardware it's time to find another motherboard.
Finally VB4 is clunky. It's pretty quick all things considered but it could be better. The next gen of VB hopefully will take the clunky out of the way it operates.