Quote:
Originally Posted by Snowhog
How and where does vBulletin create and store Search Engine Friendly Archives?
We have the store_sitemap directory in our robots.txt file as a disallow:
disallow: /store_sitemap/
I'm not sure this should be a disallow. Should the files in this directory be available to spiders/crawlers?
|
You should "allow: /store-sitemap/"; without allowing this directory to be scanned/indexed will stop Google and every other search engine from being able to download and see the site as you wish it to be seen.
Sample Page:
Quote:
<url>
<loc>http://www.domain.com /</loc>
<lastmod>2017-01-01</lastmod>
<changefreq>weekly</changefreq>
<priority>0.8</priority>
</url>
|