vbSpiderFriend - Search Engine Friendliness

I am tired of my 200,000 posts not being listed in Google. I was inspired by phpbuilder.com this morning and I wrote:

vbSpiderFriend - the search engine indexer for all of your posts

Purpose: Allow search engine spiders to crawl a linked list of all of your posts.

Project Requirements:

-Friendly URLs (no query strings)
-Good dynamic meta tags
-Never have to touch the script again.. It is Y3K compliant, simply re-submit to the engines to update your listings

Install Requirements:

-vBulletin 1.x or 2.x
-about 10 minutes


1) Download the attached Zip.

2) Open class.mysql.php and put your database login info at the top.

3) Create a new directory called archive under your forum, like /forum/archive

4) Open the included .htaccess and change the Error 404 to your new archive path.

5) Open index.php and change the self-explanatory variables at the top of the file.

6) Upload all 3 files to your archive directory.

7) Submit /forum/archive/index.php to search engines and watch em crawl


DISCLAIMER: I don't use 2.x but I checked the schema and this should work fine.

NOTES: This uses ErrorDocument and query string parsing to get the variables needed. I do not have the time or energy to troubleshoot this if it does not work on your server. Sorry!

[steven]

It appears as though the spider is only not indexing the first 2 forums that are specified for exclusion, but if there are more than 2 forums specified in the index.php file, then it still spiders the forums that I don't want to be included.

[steven]

Solved my problems, on last question. Does anybody know how often that the vbspiderfriend spiders you site? So let's say I got 100 posts on my site today and get 200 tommorrow, when does the vbspiderfriend spider those new posts?

Thanks
Steven

[Overgrow]

I have a new version that fixes all of the bugs with redirection etc, I'll try and get it uploaded in the morning. It won't require any major changes if you update, just copy over your existing variables.

Glad you got your problem sorted, maybe post the solution too so others that run into it know how to fix it as well.

re: how often does it spider?

vbSpiderFriend is not a spider itself, it is simply a new way to view your forum-- a way that is more friendly for a spider to navigate and archive. The data you see from vbSF is totally fresh and updated all the time since it reads it dynamically from the database.

The question is-- how often do you re-submit your site to the search engines? If you submit it today with 100 posts, it will always have those same 100 posts on a search engine until you re-submit. At that point, the engine spider can crawl all of your new posts. Some spiders automatically return after a certain time, some do not.

[MrLister]

on mine when i click on one of the forums it says this page cannot be found. that's when it goes to something like /archive/14 why does it do this?

[Overgrow]

Your .htaccess ErrorDocument is not pointed correctly. You MAY need to edit your access.conf to allow override all for .htaccess files.

You probably just have to edit the .htaccess and mess around with the file pointer. It needs absolute directory names most likely.

[Overgrow]

EVERYONE: Please download the latest version from the first page of this thread. Thanks!

[MrLister]

i'm using microsoft iss. how would i do it under there? if i select the file as a error thing it shows the source code and nothing else when you click on one of them.

[Overgrow]

Sorry No clue on how to do the URL trick under IIS.

[Gilby]

Originally posted by mrlister
i'm using microsoft iss. how would i do it under there? if i select the file as a error thing it shows the source code and nothing else when you click on one of them.

From http://www.microsoft.com/technet/iis/steps.asp

Custom Error Messages

In Apache, you provide custom error messages by editing the Error Document and referring to it by using the command:
ErrorDocument 404 http://www.domain.com/404.html

To customize error messages in IIS 5.0, in the IIS snap-in open Properties for the Web site. On the Custom Errors tab, you'll see the location of the error message files. From here, you can map custom error messages to a file or to a URL on the local server.

[cyrus]

hi

I would really like to get this thing working but Icant for some reason

here is the link to my arschive files

http://66.78.8.167/~thehood/archive/index.php

i keep getting 404 errors

do I need to make archives of the forums or sumthing ?? and if so, how do i make these.

thanx all !!!