vbSpiderFriend - Search Engine Friendliness

I am tired of my 200,000 posts not being listed in Google. I was inspired by phpbuilder.com this morning and I wrote:

vbSpiderFriend - the search engine indexer for all of your posts

Purpose: Allow search engine spiders to crawl a linked list of all of your posts.

Project Requirements:

-Friendly URLs (no query strings)
-Good dynamic meta tags
-Never have to touch the script again.. It is Y3K compliant, simply re-submit to the engines to update your listings

Install Requirements:

-vBulletin 1.x or 2.x
-about 10 minutes


1) Download the attached Zip.

2) Open class.mysql.php and put your database login info at the top.

3) Create a new directory called archive under your forum, like /forum/archive

4) Open the included .htaccess and change the Error 404 to your new archive path.

5) Open index.php and change the self-explanatory variables at the top of the file.

6) Upload all 3 files to your archive directory.

7) Submit /forum/archive/index.php to search engines and watch em crawl


DISCLAIMER: I don't use 2.x but I checked the schema and this should work fine.

NOTES: This uses ErrorDocument and query string parsing to get the variables needed. I do not have the time or energy to troubleshoot this if it does not work on your server. Sorry!

[michaelab]

Originally Posted by Till

In any case, no offense to the author, I would not recommend running this script. It never worked for me (although posts showed up, none were indexed due to a bug that's been fixed now though (see: status code)) and there's a ton of scripts on vB.org that do the same or maybe a better job. (I wrote one myself to get the job done.)

That's a bit harsh. This hack works fine and, unlike some similar ones, requires virtually no messing with original vB scripts. It also doesn't require the mod_mime/ForceType directives which a surprising number of hosts (including mine) don't support.

Michael.

[anonscape]

Hi,
Great hack. Very nice feature of the dynamically geterated subject and meta tag lines. I have this installed and everything seems to be working fine. Google has been by twice, gone to the archive and has gone to the depth of /archive/4 /archive 23 etc. for all of the archives. On both visits it has not proceeded deeper to index the months weeks and posts. Is this because it takes some time for googlebot to decide to move deeper in a future spidering of the site or is this because googlebot is seeing a 404 error and then not indexing further?
TIA for any info.

[michaelab]

It isn't a 404 problem, that was solved a long time ago by the script forcing the status code to 200 (OK). It took a little while for Google to fully index my forum but it now has done and gone all the way to all the posts. Very, very effective hack.

Michael.

[anonscape]

Thanks Michael for such quick feedback to my question. I will be patient and let google do its thing.

I'm REALLY glad to know that Overgrow's script will work because it is my ONLY option. Like you, my server does not allow force type directives. That is because my hosting enviornment has phpsuexe installed. Apparently this is useful for improving the security of the server but it has a horrible consequence for php scripts. When phpsuexe is installed all .htaccess php calls to apache are void therefore force type etc. lines in the .htaccess are useless. This is because php is no longer running as an apache module and apache will not handle those directives any longer. Apparently most server enviornments and hosts are going to install phpsuexe and that will render all hacks like xenon's, scuzzy's indexers, which rely on force type mime mod, features void.
Quote from "PHPSUEXEC - What do I need to know?" - "All servers will be running phpsuexec within the near future."

Overgrow's hack will be the only SE Friendly option for the majority of vBs soon.

[michaelab]

Ah - I don't know anything about phpsuexec but if it doesn't handle .htaccess directives then I'm not sure how the script works at all because allthough it doesn't require "force type" it does require the:

ErrorDocument 404 /forum/archive/index.php

(alter according to forum directory) line. If however you can hit your archive link and follow the links through in your browser then it must be working.

The ways that Google works are weird and wonderful and I don't think anyone who doesn't work there really understands them. One thing you'll need to do of course in case you haven't already is have a link to your archive directory somewhere in your forum (I hacked the footer so it appears on every page next to the "Contact Us" link). Otherwise, Google will have nothing to start with.

Michael.

[anonscape]

phpsuexec does block force type commands but allows for the .htaccess file to issue the error document line.
I put a link to the archive at the footer as well.

I checked out your forum at zerogain.com ... very nice and also noticed that it is number 1 for a google search for "vbSpiderFriend by ~shabang~ of Overgrow"

Therefore the google spider is really happy with what it finds at your archives - congrats and thanks for your help.

[vbmechanic]

Originally Posted by michaelab

That's a bit harsh. This hack works fine and, unlike some similar ones, requires virtually no messing with original vB scripts. It also doesn't require the mod_mime/ForceType directives which a surprising number of hosts (including mine) don't support.

Michael.

It was also the first spider-friendly archive for vBulletin and probably the inspiration for all the following scripts and the archive now included with vB3. Overgrow also posted a good article somewhere around here about making your archive more inviting with H1 tags, other optimizations, and more links into your site.

[michaelab]

Originally Posted by anonscape

I checked out your forum at zerogain.com ... very nice and also noticed that it is number 1 for a google search for "vbSpiderFriend by ~shabang~ of Overgrow"

Therefore the google spider is really happy with what it finds at your archives - congrats and thanks for your help.

Whoa! :surprised: I didn't expect that. I guess most people are using other spider hacks these days.

btw, useful tip for finding all Google records for your site is to use the "allinurl:" command ie, to search for all Google hits of my site I'd search for:

"allinurl:www.zerogain.com"

Then, click on the "repeat the search with the omitted results included" link to expand out everything and hey presto, all your hits

Michael.

[Thomas P]

Hello,

we are using the original hack from overgrow since it was introduced in 2001 and it does a hell of a good job
Best hack ever for vB2 imho.

I have a question regarding compatibility with vB3.

Given we upgrade from vB2 to vB3, what happens to all the archive links out there?
Will they still work?
What's the best strategy here? Put an 404.php online which redirects to the "new" archive? Would this be considered as "cloaking" by google?

I tried overgrow's hack with vB3, which can run parallel to the new archiv for compatibility reasons - seems to work.

Can anyone confirm or has someone a better idea?

Thanks,
-Tom