vbSpiderFriend - Search Engine Friendliness

I am tired of my 200,000 posts not being listed in Google. I was inspired by phpbuilder.com this morning and I wrote:

vbSpiderFriend - the search engine indexer for all of your posts

Purpose: Allow search engine spiders to crawl a linked list of all of your posts.

Project Requirements:

-Friendly URLs (no query strings)
-Good dynamic meta tags
-Never have to touch the script again.. It is Y3K compliant, simply re-submit to the engines to update your listings

Install Requirements:

-vBulletin 1.x or 2.x
-about 10 minutes


1) Download the attached Zip.

2) Open class.mysql.php and put your database login info at the top.

3) Create a new directory called archive under your forum, like /forum/archive

4) Open the included .htaccess and change the Error 404 to your new archive path.

5) Open index.php and change the self-explanatory variables at the top of the file.

6) Upload all 3 files to your archive directory.

7) Submit /forum/archive/index.php to search engines and watch em crawl


DISCLAIMER: I don't use 2.x but I checked the schema and this should work fine.

NOTES: This uses ErrorDocument and query string parsing to get the variables needed. I do not have the time or energy to troubleshoot this if it does not work on your server. Sorry!

[Jesse69]

Hi - Im very interested at the script - but cant get at the first Page. - please tell me where to get it ....

Jesse

http://161.58.84.213/forum/showthrea...threadid=15628
produces errors

[Jesse69]

Ok , seems someone fixed that prob- no I got the Hack , but :
Fatal error: Call to unsupported or undefined function htmlheader() in /homepages/2/d21873614/htdocs/forum/archive/index.php3 on line 69

any hints ?

Jesse

[Overgrow]

Hi Jesse,

You need to upgrade from PHP3 to PHP4. The script will work and everything will be faster and more pleasant.

og

[chrispadfield]

google is being incredibly slow to index us... um..

[Overgrow]

The google update has already happened for last month and the databases appear stabilized across www-www2-www3. Check next month around this time and I'm sure we'll have more VBSFs in there.

[Jesse69]

@ Overgrow

php4 ? No chance for this year at my provider ... :-(

looks like the searchengines will not found us anymore ...

Jesse

[Overgrow]

IMPORTANT MESSAGE FOR SPIDERFRIENDLY FORUMS:

This little piece of Google info came from a great interview about Google's Page Rank and Term Vectors.

...while Google might spider your site, it won't be added to the database until someone else links to it.

I was guilty of this myself with the vbSpiderFriend here. I didn't want this link to be visible to my users-- or they might start demanding a bloat-free text only version of my forum! So I manually submitted my Archive to Google hoping it would be added to the database.

According to that article (and interview with a head Goog) if a website is an island-- with no links leading into it from anywhere-- it will never be added to the database. So! Lesson learned: you must put a link to your archive somewhere on your site that Google normally indexes each month. Also, your Page Rank will of course be higher since it was found and not submitted.

[eva2000]

Originally posted by Overgrow
IMPORTANT MESSAGE FOR SPIDERFRIENDLY FORUMS:

This little piece of Google info came from a great interview about Google's Page Rank and Term Vectors.

...while Google might spider your site, it won't be added to the database until someone else links to it.

I was guilty of this myself with the vbSpiderFriend here. I didn't want this link to be visible to my users-- or they might start demanding a bloat-free text only version of my forum! So I manually submitted my Archive to Google hoping it would be added to the database.

According to that article (and interview with a head Goog) if a website is an island-- with no links leading into it from anywhere-- it will never be added to the database. So! Lesson learned: you must put a link to your archive somewhere on your site that Google normally indexes each month. Also, your Page Rank will of course be higher since it was found and not submitted.

hence i linked the my url when i installed your hack to the bottom of my forum pages - i have already been indexed too

[Brian]

How could you make this only display the months/weeks when their are posts made.. IE for not popular forums some areas dont get posted often and I would rather the search engine not get the "No posts, please go back.:" phrase so often.

[Overgrow]

Doing this would make the display time unbearable I believe. Maybe not if your forum DB is really small.. but if you want to remove the dead weeks you would have to query each week as it is displayed in the list..... adding many many queries to the week-list.

Good idea, easy to implement, but mind-boggingly slow if it's done.