You could remotely host your attachments on another server if you have two boxes. I'm doing this for bandwidth usage reasons, but it would lower your server load as well, as the queries for the attachments are being run on the remote server, and the images are linked directly from the remote server.
If you're interested, there's a hack in Beta Hacks that does this.