I have a feature request... to dump a single text file, gzipped, of ALL of the urls that go into the various sitemaps.
Basically... when we're in the loop to create the various sitemaps, to additionally write a text file, with just one full URL per line.
This is because this would also be good for Yahoo and other spiders. Yahoo specifically asks for such a thing on their submit page:
http://submit.search.yahoo.com/free/request
Quote:
You can also provide the location of a text file containing a list of URLs, one URL per line, say urllist.txt. We also recognize compressed versions of the file, say urllist.gz.
|
And to me... it seems that the loop to create the Google Sitemap, is the perfect low overhead place to also dump the Archive URL's into a text file for Yahoo and other spiders to feed from.