vbSpiderFriend - Search Engine Friendliness

I am tired of my 200,000 posts not being listed in Google. I was inspired by phpbuilder.com this morning and I wrote:

vbSpiderFriend - the search engine indexer for all of your posts

Purpose: Allow search engine spiders to crawl a linked list of all of your posts.

Project Requirements:

-Friendly URLs (no query strings)
-Good dynamic meta tags
-Never have to touch the script again.. It is Y3K compliant, simply re-submit to the engines to update your listings

Install Requirements:

-vBulletin 1.x or 2.x
-about 10 minutes


1) Download the attached Zip.

2) Open class.mysql.php and put your database login info at the top.

3) Create a new directory called archive under your forum, like /forum/archive

4) Open the included .htaccess and change the Error 404 to your new archive path.

5) Open index.php and change the self-explanatory variables at the top of the file.

6) Upload all 3 files to your archive directory.

7) Submit /forum/archive/index.php to search engines and watch em crawl


DISCLAIMER: I don't use 2.x but I checked the schema and this should work fine.

NOTES: This uses ErrorDocument and query string parsing to get the variables needed. I do not have the time or energy to troubleshoot this if it does not work on your server. Sorry!

[SoloX]

Is there any way to feed the archive URL to something like website downloaders to create a full text archive of the VBB? This way, the search engines wont see the nasty 404s. I tried using some win based site downloaders and all stopped at the first page of the archive index (due to 404's).

[SloppyGoat]

I don't think I'll try this one. "http://www.overgrow.com/edge/archive" gives me this....

Parse error: parse error, expecting `T_OLD_FUNCTION' or `T_FUNCTION' or `T_VAR' or `'}'' in /home/overgrow/HTML/globals/class.mysql.php on line 1

Fatal error: Cannot instantiate non-existent class: mysql in /home/overgrow/HTML/edge/archive/index.php on line 78

[lalo landa]

there 's an option to make work the vbcodes?

REally cool hack Thnkxs!!

[floridaideal]

Hi all

Thanks for this great hack, I installed it about 3 months ago but so far its not been spidered.

I had the link added to my index page www.top-forums.com and google has now picked the link up and has listed it as http://www.top-forums.com/forum/archive/index.php which is correct.

Its all working ok as far as I can see so why isnt it spidering the whole archives?

Someone please help.

Thanks

Stu

[Prince]

Originally posted by floridaideal
Hi all

Thanks for this great hack, I installed it about 3 months ago but so far its not been spidered.

I had the link added to my index page www.top-forums.com and google has now picked the link up and has listed it as http://www.top-forums.com/forum/archive/index.php which is correct.

Its all working ok as far as I can see so why isnt it spidering the whole archives?

Someone please help.

Thanks

Stu

Google. will not spider mine either, I have had it installed for around 4 months.

Apparently my page rank on my forums is not high enough.

[floridaideal]

Can anyone please help Ironman and me ?

Thanks

[ryancooper]

I get a "The page cannot be found" error when I click on the forum links? Any idea why??

Thanks!!

[aquariumpro]

Hi:

Since switching hosts ( a much better host!) and upgrading to VBulletin 2.30, vbspiderfriend is still working and we have thousands of links in Google and other SE. Thanks Overgrow!

However, a new development has started, which does not seem to affect the actual function of the script, but is generating hundreds of Kb a day in our server error log that look like:

[Wed Jun 25 09:00:06 2003] [error] [client 211.28.92.157] File does not exist: /home/aquariumpros/websites/www.aquariumpros.com/html/forums/archive/19/2003/01/2/785


These errors are being generated every time a user posts in our forum, and with about 100 posts a day or more, the error log reaches over 20,000 Kb a week!


What's up? and do you think it's possible to fix this?

URL: http://aquariumpros.com/forums/archive/

The script is working fine. It picks up all the new posts and can be easily browsed. Other than those error log entries, it works perfectly! Problem is that that error log bloating is running me ouyt of disk space and it must be emptied every other day.

[KevinM]

I have installed this hack on my test forum, and it works great on 15 out of 20 of my forums.

Of the ones that don't work, all the forum id's are single digit e.g.
http://www.ukgardeningchat.co.uk/archive/5 where the forum id is 5.

However, if I manually type in 05 at the end of the url instead, it works great. eg.

http://www.ukgardeningchat.co.uk/archive/05

but it does not do this automatically. All the forums from 10 upwards are fine. Any suggestions on how to make it work or force the url to read 05?

[michaelab]

05-21-01 at 05:29 PM Overgrow said this in Post #83
Streicher: Sorry I have no idea on your problem-- you are the only one who has reported this and I cannot duplicate it.

Well - I am getting exactly the same problem! To re-cap, the problem is that for forums with an ID < 10 clicking on the forum link just reloads the top level arcihive list, and doesn't go into the forum itself.

If you manually add a 0 to the forum ID in the URL (eg .../archive/06) then it loads find and what's more, all subsequent links off the forum also work fine.

I couldn't find the exact problem but for some reason the code that splits up the URL:

PHP Code:

if (ereg("/archive/[0-9]",getenv('REQUEST_URI'))) {

    $urlArray=explode("/",getenv('REQUEST_URI'));  //split the URL path
    $a=0;

    while($urlArray[$a] != "archive") {
        $a++;
    }

    $forumID=$urlArray[($a+1)];
    $year=$urlArray[($a+2)];
    $month=$urlArray[($a+3)];
    $week=$urlArray[($a+4)];
    $threadID=$urlArray[($a+5)];

    if (eregi("[a-z]",$forumID) or $forumID < 1 or strstr($privateForums,"|".$forumID."|"))
        $forumID=1;
} 


...is parsing the forum ID as nothing (empty string) for forum IDs < 10. I couldn't be bothered to work out why that was so I fixed the problem using a workaround:

In the forumList() function I changed:

PHP Code:

    if(!empty($data)) {

        while ( list ( $key,$forum ) = each ($data)) {
            echo spacer(1)."<a href=\"$baseURL/$forum[forumid]\">$forum[title]</a><br>";
        }
    } 


to...

PHP Code:

    if(!empty($data)) {

        while ( list ( $key,$forum ) = each ($data)) {
            $forumNo = $forum[forumid];
            if ($forumNo < 10)
                $forumNo = "0" . $forumNo;

            echo spacer(1)."<a href=\"$baseURL/$forumNo\">$forum[title]</a><br>";
        }
    } 


That way, the 0 is prepended to the forum ID in the output and it all works fine.

I know this thread is pretty old but hopefully it will help some of you out there!

BTW, what's the status on this Hack working with Google or not re. the 404 issue? I have to use this hack rather than any of the others because I can't seem to get the mod_rewrite or ForceType directives to work using Skuzzy or Teck's hacks.

Michael.