Google sitemap for the vB Archives. Redirect human and robots.

Release V1.2 (9 Nov 2005)
* Higher sitemap priority rate is given to threads with new posts. So Google can index fresh threads first.

* Not recommending the original optional STEP 3 hack. To avoid potential Google penalty, my advice is to remove the STEP 3 hack.

Release V1.1a (12 Oct 2005)

* Bug fix only

Release V1.1 (9 Oct 2005)

* Can handle very large forums with more than 50,000 URLs per forum
URLs will be spanned through multiple files for each large forum.

* Created a function to detect search engine crawlers. The vB built-in
search engine detector can only identify about 3 or 4 search engines.
My function will detect over 20 search engine crawlers.

* Support forums hosted by web servers that do not support 'fix_pathinfo'
ie. instead of the usual 'archive/index.php/f-10.html' link. These
forums have a link as 'archive/index.php?f-10.html'.

* Alert about wrong directory permissions to help newbies.

* Automatically write index file to archive directory if the php
script can not write into the base vB directory.

* Bug fixes.


Objectives
==============

• Create Google sitemap files and sitemap index file for vB archives, submit to Google by the Scheduled Tasks.
• To have the vB Archive used as a mirror to the actual threads.
• Google loves the nature of the archive pages, as they are static and do not contain repeated contents.
• Google gauge pages heavily based on external links. We need to redirect these external thread links to the archive pages.
• We often see vbulletin archive in the Google search results, but the users are taken to the archive page instead of the actual threads. We need to automatically redirect visitors to the actual threads instead of the archive. Otherwise the visitor either need to reclick for the Full Version or read the dull archive contents.

Q and A
==============
Q. Would the sitemap contain the links for hidden forums?
A. No, the forum permission was consulted while generating the sitemap files.

Q. How often are the sitemap files generated?
A. You decide and set in the Scheduled Tasks. The script can not be called by external user by default to prevent boring people killing your server.

Q. Is the sitemap file compressed.
A. Yes, the multiple sitemap files are gunziped according to Google sitemap standard to save bandwidth. Sitemap index file is not compressed, it is submitted as a normal xml file.

Q. Would the sitemaps include links for the normal threads? eg. showthread.php?t=1234...
A. No, it is unlikely Google will index your entire site if you feed it with all the combination of showthread links. It is better to let Google going through the more static archives. You will have a better chance for sure to have more thread contents indexed by Google this way.

Q. Why don't you go crazy about rewrite rules and do things like including thread title as the url.
A. I won't deny having keywords in the url is a good SEO strategy, but Google also does not like "Over Search Engine Optimized" web sites. Google has recently penalized a huge number of such sites. Sending them from page rank of 5, 6 to 0.

Q. Does sitemap really help?
A. Definitely, Google has done over 60,000 pages since I submitted my sitemaps a few days ago. Yahoo bots were visiting more pages than Google before the sitemap. I expect the total Google visits for this month will be exceeding Yahoo in the next one or two days.

What is involved?
==================
I have divided this hack into two steps. The first step involves unloading a php file. This enables the sitemap to be generated and submitted to Google.

The second step involves installing a Plugin using AdminCP. This sends all robots to the archive pages, preventing them viewing the actual threads.

For example, Google/Other Crawlers follows an external link to visit:
http://forums.mysite/showthread.php?t=1234&page=2

It will be told this page is permanently relocated to:
http://forums.mysite/archive/index.php/t-1234-p-2

This way you don't lose page rank gain from external links.

Install
=========
To install, follow the readme file.
To let me know you have installed this and let me send update information to you. Please click INSTALL .

Strategy
=========

It is unlikely Google/other Search Engine will index your entire site, especially due to the dynamic nature of the vbulletin forums. An archive sitemap will let Google concentrate on the real contents of your forums -- the threads. If Google needs to go through the endless member profile pages. It will get sick of it and just become tired.(sorry, perhaps robots can not become tired). What we can do is disallowing the crawling of unneccessary pages. My robots.txt contains:

#ALL BOTS
User-agent: *
Disallow: /admincp/
Disallow: /ajax.php
Disallow: /attachments/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /images/
Disallow: /includes/
Disallow: /install/
Disallow: /modcp/
Disallow: /subscriptions/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /announcement.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /external.php
Disallow: /faq.php
Disallow: /frm_attach
Disallow: /image.php
#Disallow: /index.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php?
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /payment_gateway.php
Disallow: /payments.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /usercp.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php

You perhaps have noticed I included index.php in there. Apparently Google regards http://forums.mysite/index.html as same as http://forums.mysite/
...but http://forums.mysite/index.php as a different file. The default vB templates include index.php as the internal link. That will spread your page rank on your home page! So it is better off not letting Google see this file.

If you have rewrite installed. Perhaps you could add to the .htaccess file:

RewriteCond %{QUERY_STRING} ^$
RewriteRule ^index.php$ / [R=301,L]

(if your forums are under http://site/forums/. Try: RewriteRule ^forums/index.php$ forums/ [R=301,L])

That will redirect /index.php to /, but only if no query_string is presented. ie. /index.php?do=mymod will not be redirected.

[VaaKo]

this is the error:

Warning: fopen(http://www.google.com/webmasters/sitemaps/ping?sitemap=http%3A%2F%2Fwww.oneforum.org%2Fg_sitemap.xml): failed to open stream: Connection timed out in /archive/forums_sitemap.php on line 264

Warning: feof(): supplied argument is not a valid stream resource in /archive/forums_sitemap.php on line 268

Warning: fread(): supplied argument is not a valid stream resource in /archive/forums_sitemap.php on line 27

....

[dutchbb]

Hmz not easy....

I normally assign the permission this way:

#chown apache.MYUSER_GROUP archive
#chmod 775 archive

MYUSER_GROUP is the user group my login belongs to. #ls -l will show that.

I understand chmod 755 (it was actually already set to 777), but what do you mean with: '#chown apache.MYUSER_GROUP archive' and 'MYUSER_GROUP is the user group my login belongs to. #ls -l will show that' ? That's chinese to me !

PLease explain, thank you

[lierduh]

Originally Posted by Triple_T

Hmz not easy....


I understand chmod 755 (it was actually already set to 777), but what do you mean with: '#chown apache.MYUSER_GROUP archive' and 'MYUSER_GROUP is the user group my login belongs to. #ls -l will show that' ? That's chinese to me !

PLease explain, thank you

The problem is no one wants to know/learn about the very basics of directory/file permission.

If you have 777, it means everyone can write to it. You won't need to bother with 'chown'. The example I provided was for someone who knows a bit more and wanting to have the more secure way. I will rewrite the instruction at next release and have one set of simple instruction for newbies. In the meantime, if you are still interested about directory permissions, please read:

https://vborg.vbsupport.ru/showpost....6&postcount=27

[VaaKo]

what about my error?

[lierduh]

Originally Posted by Lebanese Forces

what about my error?

It means your web server could not connect to Google at port 80. It is possible Google was down at the time, or your web server's firewall prevents the script doing so. I just tried manually:

http://www.google.com/webmasters/sit...Fg_sitemap.xml

It returned:
=========
Sitemap Notification Received

Your Sitemap has been successfully added to our list of Sitemaps to crawl. If this is the first time you are notifying Google about this Sitemap, please add it via http://www.google.com/webmasters/sitemaps so you can track its status. Please note that we do not add all submitted URLs to our index, and we cannot make any predictions or guarantees about when or if they will appear.
===============

You can always comment out those lines in the script and manually submit the sitemaps.

[buro9]

To those unsure about this hack, I would say persevere.

My number of Google spiders hasn't increased dramatically, but they're being far more efficient.

A month ago the number of pages I had in Google was only 66,000, now I have over 846,000 pages indexed: http://www.google.co.uk/search?q=site%3Awww.bowlie.com

It really is worth it, although the code edits in archive/index.php can be a bugger to get your head around at first.

[dutchbb]

Originally Posted by lierduh

The problem is no one wants to know/learn about the very basics of directory/file permission.

I do. That's why I replied:

PLease explain, thank you

The more I can lurn , the better

If you have 777, it means everyone can write to it. You won't need to bother with 'chown'. The example I provided was for someone who knows a bit more and wanting to have the more secure way. I will rewrite the instruction at next release and have one set of simple instruction for newbies. In the meantime, if you are still interested about directory permissions, please read:

https://vborg.vbsupport.ru/showpost....6&postcount=27

thank you, I will look into it and try to setup the hack

[dutchbb]

BTW I created a google sitemap account after the task. Is this needed, or do these need to be joined or something? And how does the vb task send the sitemaps if the google account wasn't even setup?

Also:

Check your web log and see if the Search Engine visits are being redirected.
You should see 301 (Permanent Redirect) for the actual thread visit and
then 200 (ok) for the archive pages.

What weblog is there in vbulletin, if you mean currently active users: i don't see it there.


Alot of newbie questions again

[PixelFx]

Ok I know chmod, and set my archive/internal files to 777 .. I still get

Warning: fwrite(): supplied argument is not a valid stream resource in /archive/forums_sitemap.php on line 247

Warning: fwrite(): supplied argument is not a valid stream resource in /archive/forums_sitemap.php on line 253

Warning: fclose(): supplied argument is not a valid stream resource in /archive/forums_sitemap.php on line 254

I also did my root as 750 ..

my direction setup is url/forum/index.php for forums, I changed your robots text to disallow: /forum/files/ etc

and I still get error above, I also setup my .htaccess etc,

I'm running vbulletin v3.5.0 gold.

any sugguestions?

[VaaKo]

Originally Posted by PixelFx

Ok I know chmod, and set my archive/internal files to 777 .. I still get



I also did my root as 750 ..

my direction setup is url/forum/index.php for forums, I changed your robots text to disallow: /forum/files/ etc

and I still get error above, I also setup my .htaccess etc,

I'm running vbulletin v3.5.0 gold.

any sugguestions?

i'm getting the same error!