Memory wise. Don't buy anything except
Corsair memory in my opinion. And get the best type your motherboard can support (check the MB specs for corsair memory). Motherboard wise, I'd buy either Asus or Gigabyte. But choose one very wisely that has the right specs to suit your needs now and in the future in you have plans to expand your system further down the line. Check things like how many HD's it can connect. Lots of motherboards now only come with one IDE slot these days. So if you want to use more than two CD/DVD-RW drives. You'll need to look for a motherboard that has two IDE slots. Becuase most HD's now use SATA connection instead of IDE. They are losing an IDE slot these days because of that. But if you get a motherboard that still has two IDE slots, it gives you the option to install 4 CD/DVD drives, while using the SATA connections for your HD's like I did. To get 4 HD's and 4 CD/DVD drives installed.
I would also consider building an SLi based system (twin graphics). But make sure your power supply and motherboard are both SLi ready. And buy the biggest power supply you can - about 650 watts. So you have the right power supply to expand in the future if you wish it to cope with the power demands of adding extra fans or drives etc. Get a good case, aluminum ones are the best because they help to cool air down faster and prevent system overheating. And good ones like SilverStone cases come with plenty of cooling fans fitted as standard. No use building a top spec system to find it's overheating all the time.
I just spent about ?2500 building my own system. I'll post some pictures later for you to have a look at to give you some ideas of what you can build for your money. Basically just plan very wisely, buy the right case to suit the motherboard first. No use buying a case that can only take up-to 2 HD's, and then installing a motherboard that can connect up-to 4 HD's etc. You'll be stuffed later on when you want to expand. You also say you don't want a large case. Think again. If you install plenty of internal hardware, a large case is a real must to give you much needed space for air circulation and cooling. Plus to let you arrange all the wires well away from the motherboard itself to improve air flow around the motherboard. Larger the better, and don't be conned into buying a cheap small case. You'll regret it later on!
I'd highly recormend using the same case I did here:
http://www.silverstonetek.com/products-tj03.htm
They also make a great SLi based 650 watt power supply to match it, which blows the air from the unit straight out the rear of the case, rather than some power units which blow the air down inside the case over the motherboard (that doesn't help cooling). Which I use here:
http://www.silverstonetek.com/products-65zf.htm
Buy Raptor Hardrives I'd say.