Spider friendly URLs

For vB 2.0

This little hackette is a quick fix to allow search engine bots to spider your threads.

Although this will allow the bots to index every thread on your site, it will not make the threads 'search engine optimized'. They will see exactly what you see when you visit your site. It simply removes the CGI bits from the URL's which prevents most search engine bots from spidering more than one level deep.

If you want a hack that allows to fully customize how the thread will look to the search engine bot, you should look at Overgrows more complete hack here.

The advantage of this hack over Overgrows is that it does not require htaccess support which can have performance issues. This could also be seen as a disadvantage though as my hack requires that you have mod_rewrite enabled on your Apache Server, whereas Overgrows method should work with just about any web host out there.

Take yer pick

[fastforward]

Originally posted by chaz7979
Doesnt google spider urls with ? now anyway?

You'll get a higher page rank with 'mysite.com/t127634.html' than if you used the normal url. Besides, google isn't the only searchengine out there.

[chaz7979]

Oh I know google isnt the only one out there but they are one of the only ones who was thought to have problems with the ? question marks.

[eva2000]

Originally posted by fastforward

I think the issue is with multipage threads. My initial release only rewrote the first page of links.

The difficulty is not so much rewriting the links, it's maintaining the other options such as 'perpage' and sort order etc.

i see.. i'm happy with first page indexing only anyway

[Prince]

Originally posted by fastforward

Besides, google isn't the only searchengine out there.

No it's not, but it's the most important search engine!

[LanciaStratos]

Originally posted by fastforward

I intend to revisit this very shortly. I've finally started to prepare for upgrading my forums to 2.2.1. It will probably be a couple of weeks though.

Hey fastforward, are you still planning on revisiting this hack?

[tamarian]

Hi folks,

I have an idea to make this hack more powerful, and require a new addition to this hack, possibly in the code.

I thought of this after I noticed I had many Netscape 4.x users, so I created new style set for them, and placed a link on the main page to "Fix Netscape errors" which simply sets their style id to use the original vB dynamic links.

This makes spiders and most surfers use the new (spider friendly) links, while those who click the "Fix link" can use original vB links in their templates.

The disadvantage, is that some users post links to other thread in spider-friendly format, that other members have trouble accessing if they have Netscape.

What I would like to do, instead, is this:

Detect if the visitor is using a browser agent (IE, Netscape, Mozilla), and if so, set their id to use a default style set (with vB's dynamic links)

If the user agent is other than (Mozilla-compatibele) then use a special style id that contain spider-friendly templates.

It initially sounded easy to me, but then I though that spider robots cannot be cookied, and any style id I assign them in my index.php code, may have to be re-checked in other parts of the code beside index.php? i.e. showthread etc.

Is there an easy way of accomplishing this?

[tpearl5]

I had to modify a couple things in the links to get everything working properly.
This may apply to 2.2.2, however I'm not positive:

In my .htaccess file I have:

RewriteEngine on
RewriteRule ^f([0-9]+)/?$ forumdisplay.php?forumid=$1&s=$2 [L]
RewriteRule ^t([0-9]+)\.html$ showthread.php?threadid=$1 [L]
RewriteRule ^t([0-9]+)-([0-9]+)\.html$ showthread.php?threadid=$1&perpage=15&pagenumber=$ 2 [L]
RewriteRule ^f([0-9]+)-([0-9]+)/$ forumdisplay.php?forumid=$1&daysprune=1000&sortord er=&sortfield=lastpost&perpage=40&pagenumber=$2 [L]
RewriteRule ^[ft]([0-9]+)/s([^/]*)/(.+)$ /$3 [L]

In forumhome templates I have:

<a href="$bburl/midiboard/f$forum[forumid]">

(midiboard is subdirectory of site)

In forumdisplay templates I have:

<a href="$bburl/f$forum[forumid]">

And in forumdisplaybit:

<a href="$bburl/t$thread[threadid].html">


Also the base href tag in the head include needed an extra trailing slash to make images work. Other then that everything seems to be okay.

[tamarian]

Originally posted by tpearl5

Also the base href tag in the head include needed an extra trailing slash to make images work.

Tpearl, we have been slaving on this trying to fix it for old Netscape browsers, and none of us cought the missing slash problem.

Thanks! :up:

[tpearl5]

Wow.. I didn't know it was a problem. Images didn't work in MSIE without that slash for me.

[tamarian]

Originally posted by tpearl5
Wow.. I didn't know it was a problem. Images didn't work in MSIE without that slash for me.

They worked fine for IE's, but Netscapes 4.7 and earlier had a problem with them.

I even started hacking the sessions and redirecting to different style id's based on user agents! Currently, registered members get the vB dynamic links, and guests get the spider-friendly ones.

I'll keep it that way, but I'll remove the link to Netscape upgrade.

That's always the case with debugging, you look hard for difficult mistakes, and they turn out to be too obvious to catch