vbSpiderFriend - Search Engine Friendliness

I am tired of my 200,000 posts not being listed in Google. I was inspired by phpbuilder.com this morning and I wrote:

vbSpiderFriend - the search engine indexer for all of your posts

Purpose: Allow search engine spiders to crawl a linked list of all of your posts.

Project Requirements:

-Friendly URLs (no query strings)
-Good dynamic meta tags
-Never have to touch the script again.. It is Y3K compliant, simply re-submit to the engines to update your listings

Install Requirements:

-vBulletin 1.x or 2.x
-about 10 minutes


1) Download the attached Zip.

2) Open class.mysql.php and put your database login info at the top.

3) Create a new directory called archive under your forum, like /forum/archive

4) Open the included .htaccess and change the Error 404 to your new archive path.

5) Open index.php and change the self-explanatory variables at the top of the file.

6) Upload all 3 files to your archive directory.

7) Submit /forum/archive/index.php to search engines and watch em crawl


DISCLAIMER: I don't use 2.x but I checked the schema and this should work fine.

NOTES: This uses ErrorDocument and query string parsing to get the variables needed. I do not have the time or energy to troubleshoot this if it does not work on your server. Sorry!

[buro9]

yeah, sorry!

what with the upgrade to 2.2.3b, i lost a few evenings whilst i upgraded and re-applied hacks for the four boards i run.

i'm busy this wekeend, but i've booked monday off work specifically to add the final touches (the indexer), i'll open a thread in beta's and place a link to it from here on moday afternoon.

so you can download and try it then

cheers

david k

[Brian]

cool

[buro9]

OK, I have created a thread in beta's.

https://vborg.vbsupport.ru/showthrea...358#post228358

Having spent the day doing nothing that I wanted, but simply upgrading everything to 2.2.4 I am fuming at vb...
I hope to find time to finish this hack just as soon as vb can release a version that is stable!

[limey]

I feel like a jackass...this hack used to work for me until I recompiled apache...

Now the index page pulls up the forums, but the archive pages end up with 404 errors.

I have tried:

-changing the httpd.conf AllowOverride All
-adding a special directory addition to my domain.conf file..
-restarted apache a bunch of times.

.htaccess is setup as
ErrorDocument 404 /archive/index.php

I dont use srm.conf or access.conf. Everything is in httpd.conf.

I recompiled apache and php last night. Maybe this has something to do with it? I doubt it though. I did recompile php without --with-apache flag and I stil ldon't know if that matters.

thanks.

[eva2000]

Originally posted by buro9


eva2000, I shall endeavour to make sure that this does not generate files for private forums. This will be perfect for you since entering private forum id's would not be possible, since the files are static. Though it should be noted that as this will generate static files... should you later turn a public forum private then you would have to delete those files manually, hence including the $forumId in the proposed folder structure.

Proposed storage:

The folder structure...

$forumpath/archive/$forumId/$year/$month/$day

For the file names...

$postId.htm

I shall start this on Tuesday next week, and hope to have it finished by Saturday next week (I'd do it sooner, but it's my birthday and this isn't that important!).

The files will be standalone and I shall develop them with vb v2.2.2 though as I shall only be accessing user, post, thread and forum (I guess... I'll have to look at the schema) this should be backwards compatible to at least 2.x boards. Though I will only be supporting the latest version at any time.

If I run into trouble or need assistance with the schema I shall let you know.

i knew there's a reason i should use email notifications

how about having the ability to rerun a script to regenerate the static html files if you later change a public forum to private ? you could have a forumid setting in the script which you can edit to either

1. remove all static html files and recreate the static files based on new forumid settings

OR

2. remove only the static html files for the forumid which when private

would be nice to be able to have the option to set the path to where the static html files are to placed

[buro9]

You should really post on the thread in the beta's forum.
I hate to have hijacked this one, which is for someone else's release.

Anyhow, deleting by forumid, threadid, or time range is definately something that could be accomodated, since it would only really mean having to delete certain folders and their contents.

Though to be honest, this would be something built after the initial release.

Phase 1 will concentrate on generating the files.
Phase 2 can be the management of them, including deletions.
Phase 3 can be any advanced formatting and/or allowing multiple templates for cross-platform archives.

Cheers

David K

[drazq]

Hi,

I just installed the hack - and it works ...

however, there seems to be a bug in it .. Not all messages are shown. Eg. in a week with 10 messages, maybe 6-7 are shown in the archive ..

any clues to why I have this problem?

- draz Q.

[buro9]

Originally posted by drazq
Hi,

I just installed the hack - and it works ...

however, there seems to be a bug in it .. Not all messages are shown. Eg. in a week with 10 messages, maybe 6-7 are shown in the archive ..

any clues to why I have this problem?

- draz Q.

Not really, it's hard to determine the reason why not all of the posts were generated... the following spring to mind:

The 7 day date range is too specific, and you calculation of 10 in a week is based upon a more broad range.
Some posts may be in hidden forums, that only administrators and mods could see.

I would really need to have a dump of your database and re-create the problem here to be able to debug thoroughly.

I have tested on a forum of 49,767 posts, and simply right click the archive folder, and the file count matched perfectly. That said, I do not have hidden forums... but the queries are based upon guest access so these should be excluded from any caching.

For a proof of concept, extend the days younger range to several thousand, this should act as a catch-all and generate for all posts ever (it takes a while!, make a cup of tea)... this way you can verify against the not-logged in users post count on the forum home page.

Cheers

David K

[drazq]

Well,

Just now I posted a new post in one of my open forums, and it didn't show up in 2002 - March - Week 3 or Week 4. It should be there right ..

- draz Q.

[pigsy]

Those folks that have installed this hack, have you seen a significant increase in traffic coming from Google?