what's the easiest way to spider?

[jdingman]

What's the easiest way to get spiders (search engines) to ONLY archive the archive portion of a site and not actually index any of the normal pages?

suggestions about this maybe? What's the best way for spiders to index or what not?

I don't want to just simply set noindex to all the pages, but is there an extension out there or a plugin that will help me with this? thanks!!

[bo0oost]

noindex will work.

[Brandon Sheley]

can I ask why on earth you would want to do this ?

seems a bit counter-productive to me.

cheers
-Bs

[bo0oost]

can I ask why on earth you would want to do this ?

seems a bit counter-productive to me.

cheers
-Bs

Saves from duplicate content.