Spider friendly URLs

For vB 2.0

This little hackette is a quick fix to allow search engine bots to spider your threads.

Although this will allow the bots to index every thread on your site, it will not make the threads 'search engine optimized'. They will see exactly what you see when you visit your site. It simply removes the CGI bits from the URL's which prevents most search engine bots from spidering more than one level deep.

If you want a hack that allows to fully customize how the thread will look to the search engine bot, you should look at Overgrows more complete hack here.

The advantage of this hack over Overgrows is that it does not require htaccess support which can have performance issues. This could also be seen as a disadvantage though as my hack requires that you have mod_rewrite enabled on your Apache Server, whereas Overgrows method should work with just about any web host out there.

Take yer pick

[tamarian]

Hi folks,

I have an idea to make this hack more powerful, and require a new addition to this hack, possibly in the code.

I thought of this after I noticed I had many Netscape 4.x users, so I created new style set for them, and placed a link on the main page to "Fix Netscape errors" which simply sets their style id to use the original vB dynamic links.

This makes spiders and most surfers use the new (spider friendly) links, while those who click the "Fix link" can use original vB links in their templates.

The disadvantage, is that some users post links to other thread in spider-friendly format, that other members have trouble accessing if they have Netscape.

What I would like to do, instead, is this:

Detect if the visitor is using a browser agent (IE, Netscape, Mozilla), and if so, set their id to use a default style set (with vB's dynamic links)

If the user agent is other than (Mozilla-compatibele) then use a special style id that contain spider-friendly templates.

It initially sounded easy to me, but then I though that spider robots cannot be cookied, and any style id I assign them in my index.php code, may have to be re-checked in other parts of the code beside index.php? i.e. showthread etc.

Is there an easy way of accomplishing this?

[tpearl5]

I had to modify a couple things in the links to get everything working properly.
This may apply to 2.2.2, however I'm not positive:

In my .htaccess file I have:

RewriteEngine on
RewriteRule ^f([0-9]+)/?$ forumdisplay.php?forumid=$1&s=$2 [L]
RewriteRule ^t([0-9]+)\.html$ showthread.php?threadid=$1 [L]
RewriteRule ^t([0-9]+)-([0-9]+)\.html$ showthread.php?threadid=$1&perpage=15&pagenumber=$ 2 [L]
RewriteRule ^f([0-9]+)-([0-9]+)/$ forumdisplay.php?forumid=$1&daysprune=1000&sortord er=&sortfield=lastpost&perpage=40&pagenumber=$2 [L]
RewriteRule ^[ft]([0-9]+)/s([^/]*)/(.+)$ /$3 [L]

In forumhome templates I have:

<a href="$bburl/midiboard/f$forum[forumid]">

(midiboard is subdirectory of site)

In forumdisplay templates I have:

<a href="$bburl/f$forum[forumid]">

And in forumdisplaybit:

<a href="$bburl/t$thread[threadid].html">


Also the base href tag in the head include needed an extra trailing slash to make images work. Other then that everything seems to be okay.

[tamarian]

Originally posted by tpearl5

Also the base href tag in the head include needed an extra trailing slash to make images work.

Tpearl, we have been slaving on this trying to fix it for old Netscape browsers, and none of us cought the missing slash problem.

Thanks! :up:

[tpearl5]

Wow.. I didn't know it was a problem. Images didn't work in MSIE without that slash for me.

[tamarian]

Originally posted by tpearl5
Wow.. I didn't know it was a problem. Images didn't work in MSIE without that slash for me.

They worked fine for IE's, but Netscapes 4.7 and earlier had a problem with them.

I even started hacking the sessions and redirecting to different style id's based on user agents! Currently, registered members get the vB dynamic links, and guests get the spider-friendly ones.

I'll keep it that way, but I'll remove the link to Netscape upgrade.

That's always the case with debugging, you look hard for difficult mistakes, and they turn out to be too obvious to catch

[tpearl5]

I just changed the link in forumdisplay_multipagenav_pagenumber

to

$bburl/t$thread[threadid]-$acurpage.html

Now I have short URL's to multi page threads

[MarkB]

Just installed this, and it works great - just two questions:

How can I fix it so Who's Online displays where a user is instead of Unknown Location: /forum/t15321/s7406f2d946ce71b342e89b3a9607ac99.html?

Also, would it be at all possible to hack this so, instead of www.yourdomain/forum/f13/ it has www.yourdomain.com/forum/chat/ (where "chat" is the forum title)? That'd be a huge improvement, I think

But, I have to say that - as it stands - this is great!

[~DC]

hmm...Im still having trouble,
my dir is www.domain.com/forums - help!

[slinky]

Mmmm... can't help you with a problem we don't know about! Also, your site's forums don't appear...

[~DC]

me? I was testing the hack and my forums are www.designerchat.net/forums

I was just using www.domain.com/forums as an example

[etom]

Is there a (nearly) final version of this ?

After reading ~140 posts there is nothing but confusion about problems with images and netscape and what ever ...

Does an "any browser" solution exists or not ?

greets

[slinky]

Many are using this without a problem, including myself.

[~DC]

Could you please paste your code?

[slinky]

The code is exactly the same as you see here.

[~DC]

ahh...damn