vbSpiderFriend - Search Engine Friendliness

I am tired of my 200,000 posts not being listed in Google. I was inspired by phpbuilder.com this morning and I wrote:

vbSpiderFriend - the search engine indexer for all of your posts

Purpose: Allow search engine spiders to crawl a linked list of all of your posts.

Project Requirements:

-Friendly URLs (no query strings)
-Good dynamic meta tags
-Never have to touch the script again.. It is Y3K compliant, simply re-submit to the engines to update your listings

Install Requirements:

-vBulletin 1.x or 2.x
-about 10 minutes


1) Download the attached Zip.

2) Open class.mysql.php and put your database login info at the top.

3) Create a new directory called archive under your forum, like /forum/archive

4) Open the included .htaccess and change the Error 404 to your new archive path.

5) Open index.php and change the self-explanatory variables at the top of the file.

6) Upload all 3 files to your archive directory.

7) Submit /forum/archive/index.php to search engines and watch em crawl


DISCLAIMER: I don't use 2.x but I checked the schema and this should work fine.

NOTES: This uses ErrorDocument and query string parsing to get the variables needed. I do not have the time or energy to troubleshoot this if it does not work on your server. Sorry!

[gmyachtsman]

Bane, read my last post above; then open your Vbulletin board and look at the address in the URL box.

http://www.influx9.com/index.php


Then go to your VBulletin Forums Admin Control panel and look at the URL there:

I don't know what it is, but it will be different.


I think that difference may have the clue you need.

I am not familiar with the code you use to change the web pages (i.e., php? action=forums), but it is really cool. Is that or is your directory structure (archive not being in the forums folder) making a problem? Too early for me to tell, though others probably could. But before wondering any more do what I just had to above.

I am far less expert than the others around here, but I hope that this may help.

Have a look at this - it may be the 'perfect' alround solution, using a samll script in a customised 404 error page:

http://vbulletin.com/forum/showthrea...threadid=21723

Here's the key stuff if you can't be bothered to read all:

that 404 is not complex, I think that's the best content management system...a lot of hosts let you do a custom 404, but most of us will have their own server/clients server for their files I think?

now the script I posted here has nothing to do with the 404 thing, the 404 is just a few lines, you just put

header("Status: 200 OK"); before any html, and then you do:

$url = explode("/",$REQUEST_URl);
$page=$url[1];

then you check if that content exists

$content = mysql_query("SELECT content FROM content WHERE page=$page");
if (!$content) header ("HTTP/1.0 404 Not Found"); (wich will override the status: 200 OK)

and then you just put the content there!! I didn't test this, but I'm planning to use this if nobody else knows a better content management system, but I believe this is the way it's done. There are peeps using this, so it does work, just not sure all my code is correct ;D

[Overgrow]

Congrats to Eva of Animeboards who now has his vBulletin spidered by Google! He has a good page rank evidently since Google went very deep on only it's first pass. Animeboards now has 11,700 new pages listed in Google with the spiderFriend!

Also to Deadbodies.org who got their spiderFriend noticed, but without much page rank, the bot didn't travel down the links yet. Hopefully next pass...

julius-- do not use invisible links.. if Google notices this, your site will be banned. Better to use a real link, let a few users see it if they must, but keep your position in Google.

Also another note in case you missed it... do not ORPHAN your archive. It MUST be listed as a link from a page that Google already knows (ie, your home page). If you have an orphan archive and submit that link directly to Google, but the bot never finds it's way there itself, your page will never be listed in the search results. Google hates orphans...


edit: changed Eva's gender to more correctly reflect the actual person

OverGrow - I haven't really had time to check exactly how you area getting this hack to work but I guess you are doing something with the '404' method I posted two posts up. I did some checking on sitepoint and found some threads by you on how you use this method for most of your site.

Do you think it would be possible to develop your hack further so that these 'perfect' URLs are not only used by search engines but users as well? ie this sort of link is used throughout your vB as standard, and not just an extra trick for getting search engine listed.

It would be really cool if you could use the "<!-- breadcrumb, nav links -->" as the URL eg this post would have a URL like:

vBulletin_Community_Forum/Customising_vBulletin/vBulletin_Code_Hacks/Releases_Version 2.x/vbSpiderFriend_ -_ Search Engine Friendliness/

These would then also be available as for use as dynamic meta keywords.

Cheers

[Overgrow]

oh man dynamic meta keywords inserted as the page name........... BRILLIANT! Google will eat it up...

I dunno when I might get around to implementing this, but I love the idea. Thanks a million, I'll be sure to mention your name if I ever publish the hack.

I do use the 404 trick for the rest of my site including user links... now when people post a link to an article that they've bookmarked, its:

http://www.overgrow.com/article/2/3

(article 2, page 3).. same thing for the FAQ...

I haven't done it with vB yet since it would require a huge overhaul for little benefit (in my eyes). OTOH, Wayne Luke has been modifying the SitePoint forums to do just that. I'm sure a search here or there will find the W.Luke threads on vB urls.

[Overgrow]

ps. after thinking it through a bit, I'm not sure if we can do the vB message title as the final part of the URL... without having to query to match the title back with the threadid when the user requests it. I was actually more excited about using it for my articles and FAQ. It would be easy to build a lookup table for those without having to run a huge query like you would for the vB.

So I don't want to sound to excited about implementing your idea with the spiderFriend.. not sure if it's possible or feasible.. but it triggered a thought of how to do it with the rest of the site.

[julius]

I've disabled html in Vb posts.
But, if someone put some html or javascript tag, with the spyderfriend they will run.
Maybie to prevent this it's better to censor in vB some dangerous words like "javascript"?

I found some threads are not in the list. Any idea?

Just wondering ...

Is the zip in the second post the latest updated version of the hack?

I noticed the date read 05-21-2001 and just wanted to make sure before starting.

=-)

[dwh]

Originally posted by Overgrow
Some spiders automatically return after a certain time, some do not.

!! That's the first time I heard this! Which spiders (or major ones anyway) crawl only once?

Nice hack btw.

[JackG]

Will this work on Windows NT ?

Has anyone tired?