what's the easiest way to spider?
What's the easiest way to get spiders (search engines) to ONLY archive the archive portion of a site and not actually index any of the normal pages?
suggestions about this maybe? What's the best way for spiders to index or what not?
I don't want to just simply set noindex to all the pages, but is there an extension out there or a plugin that will help me with this? thanks!!
|