vBSEO Google/Yahoo Sitemap Generator for vBulletin 3.6.x, 3.5.x & 3.0.x

vBSEO Google/Yahoo Sitemap Generator - vBulletin 3.5.x
Download vB 3.8.x Compatible Version Here
Download vB 3.7.x Compatible Version Here
Download vB 3.6.x Compatible Version Here
Download vB 3.0.x Compatible Version Here

Note: Version 2.5 replaces all previous versions of the vBSEO Google/Yahoo Sitemap Generator, and now includes support for all the new URL types in vBulletin 3.8.x.

Overview
============================

Providing Google, Yahoo, Ask sitemaps is an essential step towards improving your forums.

The "vBSEO Google Sitemaps" plugin provides a solution that allows your forum to be indexed by the Google (and also Yahoo and Ask) search engines faster, more completely, more effectively, and with less bandwidth consumption.

The ultimate goal is to help you achieve more web traffic for your forum.

Originally Posted by Google's Description of GSM

https://services.google.com/inquiry/publishertools?hl=en_US
Google Sitemaps is an easy way for you to help improve your site?s representation in the Google index. It's a collaborative crawling system that enables you to communicate directly with Google. You get a smarter crawl because you can provide Google with specific information about all your web pages, such as when a page was last modified or how frequently it changes.

Note: This plugin is fully compliant with vBSEO 3.x version. The current version is vBSEO 3.2. However, if you are running vBSEO, please upgrade to vBSEO 3.2 or greater before installing.

Compatibility
============================

Fully compatible with:

• vBulletin 3.0.x, 3.5.x, 3.6.x, 3.7.x and 3.8.x
• vBulletin Blogs 1.0.x and 2.0.x *Optional
• vBSEO 2.x, 3.x *Optional

What does it do?

============================

• Auto-generates a Google XML sitemap for all of your specified forum URL types (including vBulletin Blog URLs).
• Auto-generates a Yahoo text sitemap for all of your specified forum URL types.
• Auto-notifies Google when a new sitemap is generated.
• Automatically sends you a summary of the sitemap generation via email each time it is created.
• Includes only publicly accessible forums and threads.
• Allows Google/Yahoo to find all of your forum URLs without having to spider the forum to find the links.
• Allows Google/Yahoo to find your forum URLs without having to account for query string and/or session ID information.
• Allows Google to give precedence to crawling your higher priority forum pages.
• Allows Google to give precedence to crawling your forum pages that change more frequently.
• Allows you to conserve bandwidth by more efficiently facilitating the crawling processes for GoogleBot and Yahoo Slurp.
• Is 100% compatible with the static CRUs (Content Relevant URLs) used by vBSEO-enabled forums.

How does it work?
============================

Whenever it is scheduled or invoked via the URL, this plugin will generate both the Google XML formatted and the Yahoo text formatted sitemaps. If you selected "Yes" to the Google auto-notification setting, Google will be notified each time the sitemap is generated. The Yahoo sitemap must be manually submitted to http://submit.search.yahoo.com/free/request/.

• Google Sitemap files are gzipped
• Forum and thread pages are listed with pagination (i.e. all pages for each forums/thread are included in the sitemap)
• Maximum Priority is set for Sticky threads.
• Optimization for sitemap generation on large forums.

Automated Reporting Tool
============================

The vBSEO Sitemap includes a report tool, allowing you to track your progress. To access your reports enter the following URL:

http://www.example.com/forums/vbseo_sitemap/

NOTE: Replace "http://www.example.com/forums/" with the full URL to your vB forum root.


You will be required to login by providing your "Sitemap Interface Access Password" from your vBSEO Sitemap settings in:

• vBulletin Admin CP (for 3.5.x/3.6.x/3.7.x/3.8.x users) OR
• vbseo_sitemap.php configuration file (for 3.0.x users) located in the vbseo_sitemap directory in your forum root.

Once you successfully login, you will be provided with a list of 'Useful Links'. Here are the 4 main links:

Show Reports List
The 'Show Reports List' link will display a report of each time your vBSEO Sitemap generator executed, including the (1) date, (2) run time, (3) total URLs included, and (4) and change in # of URLs included from the last generation.

Each vBSEO Sitemap generation record will include a (1) "View details" and (2) "Remove record" link. The "View Details" link shows more detailed information about the sitemap generated. The "Remove record" will permanently delete the record from your reports.

Show Sitemap Downloads Log
The 'Show Sitemap Downloads Log' link will display a report of each time your vBSEO Sitemap was downloaded by either a search engine (GoogleBot or Yahoo) or by another user agent (i.e. Web browser, etc.).

The report includes the (1) date, (2) name of sitemap file downloaded, (3) Bot, (4) IP address of the downloader, and the (5) User-agent.

Each vBSEO Sitemap download record will include a "Remove record" link which will permanently delete the record from your reports.

Run Sitemap Generator
The 'Run Sitemap Generator' link will automatically regenerate your Google and Yahoo sitemaps.

Your Forums Homepage
The 'Your Forums Homepage' link is a quick link back to your forum homepage.

Additional Links
Additional links to various other related SEO resources are also included.


Click "Install"
============================

Please click "Install" to show your support for this script.


Installation & Configuration for vB 3.5.x, 3.6.x, 3.7.x and 3.8.x as a Plugin
============================

Follow the installation/upgrade instructions provided in the docs/readme.html file.


Google Webmaster Accounts
============================

It is NOT required for you to have a Google Webmaster account to run the vBulletin Google Sitemaps plugin. If you decide to setup an account for testing purposes you can do so here:
http://www.google.com/webmasters/sitemaps/login?sourceid=gsm&subid=us-et-about2


Copyright & Licensing

============================

• vBSEO Google Sitemaps plugin is copyright protected by Crawlability Inc.
• You are free to download and install this plugin on any vBulletin forum for which you hold a valid vB license.
• Unauthorized distribution of any code and/or documentation, whole or in part, included here is strictly prohibited.
• In addition, the following copyright notice must be maintained.

vBSEO(TM) ? 2005-2009 Crawlability, Inc.


Revision History
============================

02/22/2009 - Version 2.5 Released

NEW FEATURE: Blog tags URLs are included.
NEW FEATURE: Social Group Discussion URLs are included.
NEW FEATURE: Smart Priority applied for the following URL types

1. Show Post
2. Member Profiles
3. Forum Display
4. Poll Results
5. Blog Entries
6. Member Profile Albums
7. Social Groups
8. Social Groups Discussions
9. Social Group Pictures
10. Tags
11. Blog Tags

UPDATE: Compatible with vBulletin 3.8.x. Includes the following updated settings:

1. Priority Range - Show Post
2. Priority Range - Member Profiles
3. Priority Range - Forum Display
4. Priority Range - Poll Results
5. Priority Range - Show Thread
6. Priority Range - Blog Entries
7. Priority Range - Member Profile Albums
8. Priority Range - Social Groups
9. Priority Range - Social Group Pictures
10. Priority Range - Tags

Fix: Social Group Discussion database error is fixed.

Upgrade Instructions
============================

(1) Follow the installation/upgrade instructions provided in the docs/readme.html file.

(2) For versions of vBulletin 3.6.x only: Open "Schedule Manager" in the vBulletin Admin Control Panel (admincp); Remove existing scheduled task for Sitemap Generator if you have one.
Note: The new task will be automatically setup when you import the XML product file.

Other vBSEO Plugins

============================

• vBSEO - vBulletin Search Engine Optimization
  
• vBSEO Conditional Signatures for vBulletin

vBSEO Affiliate Program
============================

We welcome you to join the vBSEO Affiliate Program.

[funinthesun]

I submitted my sitemap url to google and when I went to check it's progress the next day, it said it had a 'general http error'? Any ideas what I'm doing wrong?

[Zhen-Xjell]

I highly recommend switching "yourdomain.com" to "example.com" as per the RFC documentation in your first post.

[vBSEO]

Originally Posted by funinthesun

I submitted my sitemap url to google and when I went to check it's progress the next day, it said it had a 'general http error'? Any ideas what I'm doing wrong?

Try to access the sitemap URL with your browser. If it opens successfully, your host might have had issues when googlebot attempted to download the sitemap.

Originally Posted by Zhen-Xjell

I highly recommend switching "yourdomain.com" to "example.com" as per the RFC documentation in your first post.

Changed. Thanks! I was unaware of that.

[turkforum]

iojam thank you for this great work..
I was wondering some of things about this hack..

My site stats as follows,
VB:3.5.1 Threads: 82,388, Posts: 1,320,020, Members: 139,435
Google PR: 5
Indexed Pages in Google: 500.000 (not included members)
(Once we update to vb 3.5, google has become more friendly)

If I install this hack,
Is it going to increase server load?
Estimatedly, how many pages would this help to be cached daily?
After this, are robots just going to visit the site maps?
Is it going to help earnng outbound links?

If the qestions so silly, please excuse me for my limited seo knowledge..

[iojam]

turkforum,

googlebot downloads your sitemap file(s) shortly after the submission. As for crawling/indexing of the pages included into sitemap, Google doesn't guarantee this (and of course we don't either ).
From Google Sitemaps FAQ:

Will Google crawl and index all of the URLs in my Sitemap?

We don't guarantee that we'll crawl or index all of your URLs. However, we use the data in your Sitemap to learn about your site's structure, which will allow us to improve our crawler schedule and do a better job crawling your site in the future. In most cases, webmasters will benefit from Sitemap submission, and in no case will you be penalized for it.

[T2DMan]

Are we talking about two different things? A vbseo Google sitemap, and vbseo rewrites.

A Google sitemap certainly helps Google to find your threads, and so can decrease your server load, as you are directing Google to visit specific pages more or less often. The sitemap is generated as often as you request it - either by cron, scheduled tasks or manually. There is only server load created at that instant.

(Once we update to vb 3.5, google has become more friendly) - yes, vb3.5 included many seo orientated changes, a lot of which were at my instigation. But vbulletin didnt finish the job. vbseo rewrites plus some onpage optimization, plus getting suffucient links - gives what I consider a completed seo of a forum.

Is it going to increase server load? -vbseo rewrites does increase server load, and on some servers and large forums this has been an issue. But with large forums some rewrites can be turned off, and tweaks made to get load to acceptable levels.

Estimatedly, how many pages would this help to be cached daily? - never any gaurantees about anything. But when you have clean url's, no duplicates, etc at least Google/yahoo are given the opportunity to find url's easier.

After this, are robots just going to visit the site maps? Google uses the sitemaps as a url list to then visit the actual url's. It also spiders url's that are not on the sitemaps, url's that it finds via normal spidering around the web.

Is it going to help earnng outbound links? - if your pages are able to be found on Google/Yahoo, then you are more likely to get people linking to you. So sitemaps will help pages get cached faster, and if they are seo'ed well etc then they should rank well...

[DefenceTalk]

I installed the hack, clicked install. Thanks.

Now, it is yet to be seen how it helps the forum. How can we measure the results and see if the sitemap is really working?

regards,

aaron

[caliman]

Hi Oleg -
Is the spider log coming soon?

[iojam]

caliman,

it is ready in a new vBSEO Sitemap Generator v1.3

# 11/21/2005 - Version 1.3 Released
(1) FIX: Authentication issue on .../vbseo_sitemap/index.php when forums are closed.
(2) UPDATE: "Spider Activity Log in your Forums" report added. If you have our vBSEO (2.2.1 or greater) Plugin installed, this feature will allow you to log spider activity on your forum. Then it can be automatically displayed within your Sitemap reports (found at vbroot/vbseo_sitemap/ on your forum once you install).

[Doc Great]

Yeah great, now the spider log is working. Thanks for your update.