vbSpiderFriend - Search Engine Friendliness

I am tired of my 200,000 posts not being listed in Google. I was inspired by phpbuilder.com this morning and I wrote:

vbSpiderFriend - the search engine indexer for all of your posts

Purpose: Allow search engine spiders to crawl a linked list of all of your posts.

Project Requirements:

-Friendly URLs (no query strings)
-Good dynamic meta tags
-Never have to touch the script again.. It is Y3K compliant, simply re-submit to the engines to update your listings

Install Requirements:

-vBulletin 1.x or 2.x
-about 10 minutes


1) Download the attached Zip.

2) Open class.mysql.php and put your database login info at the top.

3) Create a new directory called archive under your forum, like /forum/archive

4) Open the included .htaccess and change the Error 404 to your new archive path.

5) Open index.php and change the self-explanatory variables at the top of the file.

6) Upload all 3 files to your archive directory.

7) Submit /forum/archive/index.php to search engines and watch em crawl


DISCLAIMER: I don't use 2.x but I checked the schema and this should work fine.

NOTES: This uses ErrorDocument and query string parsing to get the variables needed. I do not have the time or energy to troubleshoot this if it does not work on your server. Sorry!

[Learner29]

> You need to put an .htaccess file like this in the /forums/archive folder:
> ErrorDocument 404 /forums/archive/index.php
> Change forums to the correct folder name for you board directory. This is
all that's required along with the index.php and the class.mysql.php
files.

thank you so much for trying to help.
this .htaccess file in the "archive" subfolder of the "vb" folder is
something I triple checked.
The file is there, and it is working, and this is why when clicking on any
of those broken links generated on the /vb/archive/index.php page, the page
itself is reloaded, for it is itself the 404 page.

[patvdv]

Originally posted by mvigod
Has anyone ever witnessed Google going more than one level deep? Since they get the 404 respose code with this hack they don't index those pages or go any deeper. They are not in the habit of indexing "Not Found 404 error" pages or following the links in them. They will follow the first set of links from the index.php but each of those really doesn't exist so the 404 error is the end of the line even though the pages have content.

I think this has to be rewritten with mod rewrite to rewrite the requests for all pages back to index.php so a 200 OK response code is given and it will be archived.

With all these posts didn't anybody realize this?

mvigod, I think you are right about this. Alot of webrippers will also trip over this. I know WebWhacker does handle the 404. Does anybody can give some input as to make this work with mod rewrite?

[Learner29]

Originally posted by patvdv, quoting above
...
I think this has to be rewritten with mod rewrite to rewrite the requests for all pages back to index.php so a 200 OK response code is given and it will be archived.

Patvdv and everyone else,

see what overgrow mentioned in the second post of this thread, about updating this..... look at the last of the fixes he mentioned.

Originally posted by Overgrow
...

FIXES:

v1.1b release

-Added header("Status: 200 OK"); for the really picky engines

is not that the 200OK response you are talking about ???


__________________________

I am trying to help here...... waiting for some courageous person to step forward and help me and others having problems to make this work.

I described my problem above, please read and try to help.

[Destee]

Oh My Gosh Overgrow !!!!! I am using this !!!!!

I have known about this thread for a very long time but never tried it because I thought I'd not be able to do it !!! Just thought that ... no real reason why ... but anyway ... ... I've just installed it ... PAINLESS ... BEAUTIFUL ... THANK YOU !!!!

http://destee.com/forums/archive

Whew ... Wow ... I missed out for a long time ... for no reason!

Thanks Again.

Destee

[Learner29]

well, if you have been that successful, would you kindly help your fellows a
bit ???

Please read me above, I have described my problem in detail.

[Destee]

Hi Learner29 ... it would be my pleasure to help you ... unfortunately, I don't have any tips to provide. I followed Overgrow's instructions step by step and it worked for me. Well ... as I think about it ... I did chmod 777 .htaccess ... which was not in his instructions. Perhaps this will help? Let us know.

Destee

[Learner29]

Detsee, that is so kind of you. I really appreciate your help.

well,

I hope overgrow would have a minute to help here.

mode of `.htaccess' changed to 0777 (rwxrwxrwx)

and

mode of `archive' changed to 0777 (rwxrwxrwx)

and

mode of `class.mysql.php' changed to 0777 (rwxrwxrwx)
mode of `index.php' changed to 0777 (rwxrwxrwx)

and still......... I have only index.php and no one single folder there...

I am getting pessimistic about this hack... frankly, I do not have the impression many people do really care to help.

[Destee]

Hi Learner ... sorry it's not working for you. If it is any consolation, vB3 will have something similiar to this hack included in it. So you won't be without this ability for long.

Don't be too discouraged, I happen to know that this original hack predates vb.org (which means he's probably given a lot of support since its creation), and I don't know if you've visited Overgrow's site before ... but it is humongous ... which I'm sure keeps him pretty busy ... even if he has nothing else to do, which I doubt.

Stay encouraged ... this really is a great community with lots of folk willing to help.

Destee

[Learner29]

I thank you a million my friend for your very kind answer.
I just wish anybody can help me get this to work.

[grooveh]

I am wondering if anyone can help me with this, please? I have seemingly installed everything as per directions, but still have a few errors, and cant figure out why?

Link to Archive.php

Is it me? Please let me know. I'm thoroughly stumped.

TIA