Spiders are beating the heck out of my server.

[Guest210212002]

Just looking for any advice here. Spiders are dominating the MySQL process on my box - at any given time I have 15-20 different spiders online, and at times it doubles my online user/query count.

Is there a decent way to lessen their load, while at the same time keeping the "benefits" of having them crawl my site? I really don't want to bump up my monthly server cost to accomodate engine rankings.

Thanks in advance. ;)

[Reeve of shinra]

<a href="http://www.theadminzone.com/forums/showthread.php?t=19465" target="_blank">http://www.theadminzone.com/forums/s...ad.php?t=19465</a>

[calorie]

Oldie but might still be goodie: here (https://vborg.vbsupport.ru/showpost.php?p=609808&postcount=19)

[S@NL - BlackBik]

Check the robots.txt tutorial (http://www.searchengineworld.com/robots/robots_tutorial.htm).

[Guest210212002]

Great info - thanks very much all. :D