vBulletin Google Site Map

There are a few vBulletin Google Site maps around, but none as comprehensive as this that I have seen. This vBulletin Google site map is based on Kalius's work on discussed on his site and at Digitalpoint

FYI, the vBulletin Google Site Map modification contains at least one potential remote file include exploit. You are hereby notified that you should uninstall this modification, including the removal of the vBulletin Google Site Map PHP files from your server. The author of the modification has been notified, and the vBulletin Google Site Map thread has been moved to the Modification Graveyard forum.

Best,
vBorg Staff

vBulletin Google site map
with ability to add extra urls
(c) by T2DMan - Michael Brandon

8Nov05+13hrsGMT - update. Added support for Yahoo Sitemap & bug fix
7Nov05+13hrsGMT - update. Thanks for your help.

I have attached 1 version with 2 different preset configs for your convenience. vBSEO and non vBSEO configs.

Configure in the vbgsitemap-config.php file.
$base=""; // "" if in root, $base="forum/"; //if in the subdirectory forum

It now works well with forum permissions, and the vBulletin's scheduled task system. I have personally installed the reviewed vBGSiteMap on a site with over 200,000 urls. Copes with the large number of urls with no problems.

The vBGSiteMap works well with vbSEO, and it lists all the old url's to allow Google to spider the old url's and so find the 301 redirects to the new, and so delete the old url's from its index.

Compared to the previous versions

• Sitemap files now written into their own directory - means this can be safely chmod 777 - thanks Sarah King
• config and class files separated out
• new help file written
• many corrections made to the code - amazing what sitting on a piece of code can do, and amazing what you can learn from personally installing on many different servers.
• Can now work on mysql versions from 3.23.58 to 5+
  

Specifications

• works with v3.03 - 3.5
• access to this file is password protected

the same password as the database via the vBulletin includes/config.php file

• creates an index file and multiple gzipped site maps to less than the maximum permitted by Google (I have in fact restricted it to 3k url's, rather than 50k as my browser takes a while to process the file when checking it)
• creates a gzipped file containing the urls for Yahoo Sitemap

• the site map files are created in the /vbgsitemap/files directory which needs to be chmod to 777

adds

• forums - with paginations of forums (excluding redirects and password protected forums)
• threads - with pagination of threads (excluding invisible threads=soft deleted, and threads inside password protected forums)
• archive forum pages with paginations
• archive versions of threads with paginations
• members

extra

• extra urls from other program types
• add a new url per line onto file.txt, and a gzipped file will be created for it, and added to the site index file.

How to install

• unzip the files
• copy the vbgsitemap folder to your root directory
• chmod the vbgsitemap/files directory to 777
• copy vbgsitemap.php to your root directory
• copy the includes/cron/vbgsitemap-cron.php file to your vbulletin includes/cron/ directory
• look over the configuration options on the /vbgsitemap/vbgsitemap-config.php file
  • $base="" or "forum/" - if your forum is the actual root, or in the forum directory etc.
  • $addthreadarchive=false; - whether you want the archive version of the threads to be added or not.
  • the rest should not need to be changed
  • change to $xtrafile="file.txt"; if you have additional urls that you want included from another package, get a list of those url's and add them to file.txt.
• Then access the /vbgsitemap/index.php file in your browser to get things happening.
• The username/password is the username/password of your database.
• Add the resulting one index file to Google sitemaps: http://www.yourdomain.com/vbgsitemap.php
• Add the other file to Yahoo sitemaps
  http://www.yourdomain.com/vbgsitemap/files/urllist.gz
• A cron job can be setup in the vbulletin scheduled tasks
  
  
  • ./includes/cron/vbgsitemap-cron.php
  • depending on the number of posts/new threads to your forum, set to run daily or weekly etc. The cron will automatically ping Google and return success/failure into the cron log.

[David_R]

There is one problem/bug in script, i have set our forums threads to be visible only for current month to all guests and members.
but the script goes on calculating number of pages on all forums ?

[David_R]

Fatal error: Maximum execution time of 60 seconds exceeded in /home2/gamers/public_html/vbsitemap-multi.php on line 422

uh, i have a large site with more then 11,00,000 threads

[krohnathlonman]

add this to the top of the script: @set_time_limit(30000);
do NOT change your php.ini

[turkforum]

Everything went so smooth
I submit the XML file to google sitemap via my account.

For the Sitemap.xml file I got an error:

Invalid date

Here is the explaination for the error.

Your Sitemap contains one or more invalid dates. This could be because a date is in the incorrect format, or the date itself is not valid. Dates must use ISO 8601 encoding, although you can omit the time portion. For instance, the following are both valid:

2005-02-21
2005-02-21T18:00:15+00:00
Dates that are in the correct format may be invalid if they specify a future date or specify a date that does not exist (for instance 2005-15-40).

Below is how it shows date at the xml file:
<lastmod>2005-09-05T02:21:43</lastmod>

[krohnathlonman]

I'm getting this too on all of mine...

there aren't any time checks currently and theres quite a few times circumstances where the time doesn't display right.

[jmw]

I got rid of it. This thing messed up my vb.

[Mr Chad]

Please fix -_-

[T2DMan]

Originally Posted by David_R

There is one problem/bug in script, i have set our forums threads to be visible only for current month to all guests and members.
but the script goes on calculating number of pages on all forums ?

How does the above policy work? Google in general does a deep crawl irregularly - ie monthly, and then the threads are available until they are removed/changed when Google gets a no permission error.

Google also does not even download the sitemap daily, and once spidered, takes a day or so for them to appear on Google, and a few days to start ranking high, especially for more competitive phrases.

I like Google to have the max threads possible, so I can get the max traffic possible when each thread is optimized. One month is not long enough for even Google PR to be assigned.

So despite the above, how would you want the site map generated? For monthly use, you would probably want only the previous two weeks site mapped. Please specify and I will program in. The archive version of the url's would also need the calculation done.

[T2DMan]

Originally Posted by turkforum

Invalid date

Here is the explaination for the error.

Your Sitemap contains one or more invalid dates. This could be because a date is in the incorrect format, or the date itself is not valid. Dates must use ISO 8601 encoding, although you can omit the time portion. For instance, the following are both valid:

2005-02-21
2005-02-21T18:00:15+00:00
Dates that are in the correct format may be invalid if they specify a future date or specify a date that does not exist (for instance 2005-15-40).

Below is how it shows date at the xml file:
<lastmod>2005-09-05T02:21:43</lastmod>

On the face of it, the date is correct per the Google examples.

So all I can consider is that the date is a future date, and should have the +00:00 number of hours set.

I will make sure that all last mod dates have the gmt set.

[T2DMan]

Originally Posted by libertate

Throwing it in the root is not clean. Why can't it go into its own folder off of forum?

Google requires it in the root, however, there is a workaround that will be in the next release. All sitemap fiels will be in their own directory, and separate from the sitemap php files.

[QUOTE=libertate]I also had the double appending problem with the domain name. www.site.comwww.site.com within the XML files.[quote]

If you don't do the setup correct, the url's are wrong. But all it needs is a browse at the xml file, and the setup corrected.

Originally Posted by libertate

Take the db and pass from vB, and don't ask for it. This is required anyway if it's a cron job.

Most variables are as per the config, only username/password and ping are required on the html entrance page.

Google wants an http:// after <LOC> which is not create if forgotten from the config.

The default in the config is $http="http://"; Take the risk and change it and see what happens as above.

Also there are some problems with the "archive" folder where it appends after the file.
Like this:
www.example.com/forum/archive/index.php/f-1.html

The exact format is customisable in the config. Please let me know your exact problem - the above is the correct default format, and you can change it for your specific instance.