Board Optimization - vBSEO Search Engine XML Sitemap

vBSEO Search Engine XML Sitemap 3.0
We are extremely pleased to inform the vBulletin.org community that a fully revamped version of the vBSEO Search Engine XML Sitemap (previously "vBSEO Google/Yahoo Sitemap Generator") v.3.0, is being released to all licensed vBulletin users effective immediately!

Note: The NEW vBSEO Search Engine XML Sitemap 3.0 replaces all previous versions of the "vBSEO Google/Yahoo Sitemap Generator", and includes support for all the new URL types in vBulletin 4.x.

New in the vBSEO Search Engine XML Sitemap v.3.0
The new sitemap alters the way data is structured in fundamental ways. Sitemap 3.0 segments different content areas into separate files, so you end up with a unique set of files per content type: forums, blogs, cms, groups, albums, members, tags, etc. In addition, the new sitemap has an extended priority feature that utilizes the built in 'weight' tool of the vBulletin XML sitemap generator for individual forums, CMS sections and articles to act as a multiplier.

We believe these changes are responsible for delivering increased returns for a number of forums from customers and testers we've been tracking. Please track your own progress via your Google Webmaster tools account, Google Analytics or Gravity Insights.

A Clean and Beautiful Interface
The new sitemap also features a clean and beautiful interface, built from the ground up. We believe it reflects the quality, precision and attention to detail placed into each aspect of this release. We are passionate about building software, and great software is beautiful inside, and out.

Overview
============================

The "vBSEO Search Engine XML Sitemap" plugin provides an easy to use solution that allows your forum to be indexed by all major search engines, including Google, Yahoo, Bing, and Ask, faster, more completely, more effectively, and with less bandwidth consumption.

The ultimate goal is to help you achieve more web traffic for your forum.

Originally Posted by Google's Description of GSM

https://services.google.com/inquiry/publishertools?hl=en_US
Google Sitemaps is an easy way for you to help improve your site?s representation in the Google index. It's a collaborative crawling system that enables you to communicate directly with Google. You get a smarter crawl because you can provide Google with specific information about all your web pages, such as when a page was last modified or how frequently it changes.

Note: This plugin is fully compliant with vBSEO 3.5.x versions. The current version is vBSEO 3.5.2. If you are running vBSEO, please upgrade to vBSEO 3.5.2 (or greater) before installing for optimal results.

Compatibility
============================

Fully compatible with:

• vBulletin 3.8.x, 4.0.x, 4.1.x
• vBulletin Blogs *Optional
• vBulletin CMS *Optional
• vBSEO 3.5.2 + higher *Optional

What does it do?
============================

• Auto-generates an XML sitemap for all of your specified forum URL types (including vBulletin Blog & CMS URLs).
• NEW! Groups forum areas into individual sitemaps grouped by section type for greater flexibility, for example:
  • sitemap_forum_X.xml.gz
  • sitemap_tags_X.xml.gz
  • sitemap_cms_X.xml.gz
• Auto-notifies Search Engines when a new sitemap is generated (optional)
• Automatically sends you a summary of the sitemap generation via email each time it is created (optional)
• Includes only publicly accessible forums and threads and includes the option of not including any specific public forums
• Allows Search Engines to find all of your forum URLs without having to spider the forum to find the links. Great for Deep Page linking!
• Allows Search Engines to find your forum URLs without having to account for query string and/or session ID information.
• Allows Search Engines to give precedence to crawling your higher priority forum pages.
• Allows Search Engines to give precedence to crawling your forum pages that change more frequently.
• Allows you to conserve bandwidth by more efficiently facilitating the crawling processes for GoogleBot, Yahoo Slurp, Ask's Teoma, Bing's msnbot, and other search engine bots.
• Is 100% compatible with the static CRUs (Content Relevant URLs) used by vBSEO-enabled forums.
• Is 100% compatible with the 'Friendly URLs' offered default with vb4.x.

How does it work?
============================

Whenever it is scheduled or invoked via the URL, this plugin will generate an XML formatted that search engines are able to read and parse. If you selected "Yes" to the Auto-notification setting, The 4 major search engines will be notified each time the sitemap is generated.

• Google Sitemap files are gzipped for space savings
• Forum and thread pages are listed with pagination (i.e. all pages for each forums/thread are included in the sitemap)
• Maximum Priority is set for Sticky threads.
• Optimization for sitemap generation on large forums.

Automated Reporting Tool
============================

The vBSEO Sitemap includes a report tool, allowing you to track your progress. To access your reports enter the following URL:

http://www.example.com/forums/vbseo_sitemap/

NOTE: Replace "http://www.example.com/forums/" with the full URL to your vB forum root.


You will be required to login by providing your "Sitemap Interface Access Password" from your vBSEO Sitemap settings in:

• vBulletin Admin CP -> vBSEO -> vBSEO Sitemap Settings
  

Once you successfully login, you will see a number of tabs. Here are the 4 main links:

Reports List
The 'Show Reports List' link will display a report of each time your vBSEO Sitemap generator executed, including the (1) date, (2) run time, (3) total URLs included, and (4) and change in # of URLs included from the last generation.

Each vBSEO Sitemap generation record will include a (1) "View details" and (2) "Remove record" link. The "View Details" link shows more detailed information about the sitemap generated. The "Remove record" will permanently delete the record from your reports.

Downloads Log
The 'Show Sitemap Downloads Log' link will display a report of each time your vBSEO Sitemap was downloaded by either a search engine (Identified by the Bot's name) or by another user agent (i.e. Web browser, etc.).

The report includes the (1) date, (2) name of sitemap file downloaded, (3) Bot, (4) IP address of the downloader, and the (5) User-agent.

Each vBSEO Sitemap download record will include a "Remove record" link which will permanently delete the record from your reports.

Search Engine Bots Activity Log
Use it to track Search engine activity on your sitemaps. This area displays how often, and what volume of pages search engines visit your site on a daily basis.

Run Generator
The 'Run Sitemap Generator' link will automatically regenerate your vBSEO Search Engine XML Site sitemaps.

Additional Links
Additional links to various other related Search Engine/SEO resources are also included.

Click "Install"
============================
Please click "Mark as Installed" to show your support for this script.

Installation, Upgrade, & Configuration
============================
Follow the installation/upgrade instructions provided in the docs/readme.html file.

Webmaster Accounts
============================
It is NOT required for you to have a Webmaster account on the search engines' websites to run the plugin, however, we recommend everyone set one up for tracking purposes.

• Google Webmaster Central. Get data about crawling, indexing and search traffic. Increase traffic to your site.
• Bing Toolbox
• https://siteexplorer.search.yahoo.com/

Copyright & Licensing
============================

• vBSEO Search Engine XML Sitemap plugin is copyright protected by Crawlability Inc.
• You are free to download and install this plugin on any vBulletin forum for which you hold a valid vB license.
• Unauthorized distribution of any code and/or documentation, whole or in part, included here is strictly prohibited.
• In addition, the following copyright notice must be maintained.

vBSEO(TM) ? 2010 Crawlability, Inc.

Revision History
============================

11/11/2010 - Version 3.0 Released
Compatible with vB3.7.x+ and vB4.x+
NEW FEATURE: All CMS URLs are supported
NEW FEATURE: Sitemap Groups by content type
NEW FEATURE: Enhanced layout and design of the Sitemap Generator Control Panel
NEW FEATURE: XSL Style sheet of the xml files for easier human reading
NEW FEATURE: Extended Priority. Utilizes the built in 'weight' tool of the vBulletin XML generator for all content areas (Forums, Blogs, CMS, Albums, etc.) to act as a multiplier. (vb4 Suite only)

Other vBSEO Plugins
============================

• vBSEO - vBulletin Search Engine Optimization

vBSEO Affiliate Program
============================
We welcome you to join the vBSEO Affiliate Program.

[bzcomputers]

Don't confuse this product with vBSEO. This is vBSEO Sitemap (a separate and free mod) it has officially been unsupported for a long time.

About the only place you are going to get support on it now is right here from fellow vB users. If you have a question or issue just ask. There are more people familiar with the product here than almost anywhere else.

[Kirk93]

I've only just noticed that this mod doesn't work for me. According to the logs, there is no bot activity on my site (even know I see them on "Who's Online"); how do I fix this?

Are you supposed to manually submit vbseo_sitemap/data/sitemap_forum_1.xml.gz etc to Webmasters?

Also, which ones should you submit?

[bzcomputers]

Originally Posted by Kirk93

I've only just noticed that this mod doesn't work for me. According to the logs, there is no bot activity on my site (even know I see them on "Who's Online"); how do I fix this?

Are you supposed to manually submit vbseo_sitemap/data/sitemap_forum_1.xml.gz etc to Webmasters?

Also, which ones should you submit?

Yes, you need to submit the index file ONLY to search engines (namely Google & Bing). Submitting the other files won't hurt but there will also be no benefit. The other files are linked through the index file and will be automatically scanned as long as the index file is submitted to the search engines.


Submittion url should look like:

http://www.yoursitename.com/forums/sitemap_index.xml.gz

Once submitted the first time, search engines will know where it is and continually look for it and search for changes/updates. You can also set up VBSEO Sitemap to automatically notify search engines each time it is updated (it is a setting). This may get your site updates indexed a few hours quicker depending on how often your site is already being scanned by the search engines. The auto-notify currently only works for Google & Bing. They are the two largest search engines anyway and where probably 90+% of your search engine traffic is coming from, so it is a handy feature.

[Kirk93]

I'm getting errors with this.

sitemap_index.xml.gz is located in /vbseo_sitemap/data/..., but when I link to it the search engines can't find the additional xml's because they're reportedly supposed to be in the root folder.

How do I fix this aside from manually adding each one? Thanks for your help by the way.

[bzcomputers]

Originally Posted by Kirk93

I'm getting errors with this.

sitemap_index.xml.gz is located in /vbseo_sitemap/data/..., but when I link to it the search engines can't find the additional xml's because they're reportedly supposed to be in the root folder.

How do I fix this aside from manually adding each one? Thanks for your help by the way.

You are correct the actual file locations for all sitemaps are in /vbseo_sitemap/data/ but that is not where you link to them. You link to the index file in the forum root.

http://www.yoursitename.com/forums/sitemap_index.xml.gz

The index sitemap file does not reside there but it and all other sitemap files are accessed there. For this to work correctly, this line should be in your forum root .htaccess .

RewriteRule ^((urllist|sitemap_).*\.(xml|txt)(\.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]

[Delfi_r]

Where is the information to run on 4.21? I uninstalled the plugin as it ceased to work and I understand that should run if I make some changes.

[Kirk93]

Originally Posted by bzcomputers

You are correct the actual file locations for all sitemaps are in /vbseo_sitemap/data/ but that is not where you link to them. You link to the index file in the forum root.

http://www.yoursitename.com/forums/sitemap_index.xml.gz

The index sitemap file does not reside there but it and all other sitemap files are accessed there. For this to work correctly, this line should be in your forum root .htaccess .

RewriteRule ^((urllist|sitemap_).*\.(xml|txt)(\.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]

Hello,

That rewrite rule was already present in HT access, but Google is still throwing up errors when I try to submit "sitemap_index.xml.gz" as a sitemap.

Any ideas/suggestions?

[bzcomputers]

Originally Posted by Kirk93

Hello,

That rewrite rule was already present in HT access, but Google is still throwing up errors when I try to submit "sitemap_index.xml.gz" as a sitemap.

Any ideas/suggestions?

Message me your site address and I'll see if I can't track down the issue.

[Easelm]

I have had this installed for a few months now and it's working great. The only thing
I don't receive is the SE Bots Activity Log, it's just blank.. Any ideas?

(I have the robots.txt added along with the meta tag for it)

I get plenty of spider bots on the site, just nothing in the log. Any suggestions or help
will be greatly appreciated

[fxdigi-cash]

There is something strange in here showed up after my web hosting "Bluehost" had been down for almost 1 day. I get this error:

PHP Code:

The component "Zend Optimizer" is not installed on the Web Server and therefore cannot service encoded files. Please download and install the Zend Optimizer (available without charge) on the Web Server. 


as if my web hosting doesn't support Zend optimizer, but I could see the feature is ON.

I use PHP 5.4 (FastCGI) and I'm wondering if everyone is the same or different on this!!

any idea ??

Update: I had to copy some zend opimizer url path from php5.2 fast-cgi to php5.4 fast-cgi and it works perfect!!