Search Engine Friendly Archives and store_sitemap directory

[Snowhog]

How and where does vBulletin create and store Search Engine Friendly Archives?

We have the store_sitemap directory in our robots.txt file as a disallow:

disallow: /store_sitemap/

I'm not sure this should be a disallow. Should the files in this directory be available to spiders/crawlers?

[bzcomputers]

Originally Posted by Snowhog

How and where does vBulletin create and store Search Engine Friendly Archives?

We have the store_sitemap directory in our robots.txt file as a disallow:

disallow: /store_sitemap/

I'm not sure this should be a disallow. Should the files in this directory be available to spiders/crawlers?

You should "allow: /store-sitemap/"; without allowing this directory to be scanned/indexed will stop Google and every other search engine from being able to download and see the site as you wish it to be seen.

Sample Page:

<url>
<loc>http://www.domain.com /</loc>
<lastmod>2017-01-01</lastmod>
<changefreq>weekly</changefreq>
<priority>0.8</priority>
</url>

[Snowhog]

Thank you. I had already made up my mind that it shouldn't be a Disallow and removed it (as an entry) from our robots.txt file.