AMD Athlon X2 Dual Core 5000+
This is the start of your problems. An
Athlon is a processor designed for desktop computers, not for servers. Unfortunatly there are a lot of hosting companies that offer this processor for (cheap) servers.
The 2nd bottleneck might be your disks. SCSI disks in a good RAID configuration will outperform the single (assuming you have just 2 single SATA disks) disk.
I really doubt if the memory (RAM) is the bottleneck.