Spider friendly URLs

For vB 2.0

This little hackette is a quick fix to allow search engine bots to spider your threads.

Although this will allow the bots to index every thread on your site, it will not make the threads 'search engine optimized'. They will see exactly what you see when you visit your site. It simply removes the CGI bits from the URL's which prevents most search engine bots from spidering more than one level deep.

If you want a hack that allows to fully customize how the thread will look to the search engine bot, you should look at Overgrows more complete hack here.

The advantage of this hack over Overgrows is that it does not require htaccess support which can have performance issues. This could also be seen as a disadvantage though as my hack requires that you have mod_rewrite enabled on your Apache Server, whereas Overgrows method should work with just about any web host out there.

Take yer pick

[eva2000]

Originally posted by fastforward

Yay...

Thanks mate.

I'll have to fix the docs and revert my templates tomorrow then.

thanks... one thing though with gzip and mod_gzip enabled i'm finding the shorter urls not being compressed on access compared to the real urls although it's being reported as gzipped at http://leknor.com/code/gziped.php

[fastforward]

Originally posted by eva2000
one thing though with gzip and mod_gzip enabled i'm finding the shorter urls not being compressed on access compared to the real urls although it's being reported as gzipped at http://leknor.com/code/gziped.php

uhh..oh! You know, I actually did wonder if that would be a problem but I checked with leknor and as you mentioned that says it is gzipped. What are you using to determine it's not?

I seem to remember that when I first implemented this it seemed slower to load. That's what prompted me to check with leknor.com. I hope I can fix this. non-gzipped pages are not acceptable

[eva2000]

Originally posted by fastforward

uhh..oh! You know, I actually did wonder if that would be a problem but I checked with leknor and as you mentioned that says it is gzipped. What are you using to determine it's not?

I seem to remember that when I first implemented this it seemed slower to load. That's what prompted me to check with leknor.com. I hope I can fix this. non-gzipped pages are not acceptable

simple browser/eye test

i loaded the shorter url page right click page properties

then loaded the real url and right click page properties and compared the 2 sizes

shorter url was 129kb and real url was 19kb

but both reported as 19kb when tested at leknor

[eva2000]

oh yeah another thing i;m using the Zend cache trial so with html pages they won't be cached right ?

[fastforward]

Originally posted by eva2000
simple browser/eye test

i loaded the shorter url page right click page properties

then loaded the real url and right click page properties and compared the 2 sizes

shorter url was 129kb and real url was 19kb

but both reported as 19kb when tested at leknor

I don't trust the browser method.

When I do that on threads I get 81K for the short ones and not available for on the originals. And for the forumhome I get 56K with the short url and 58K with the long url.

[fastforward]

Originally posted by eva2000
oh yeah another thing i;m using the Zend cache trial so with html pages they won't be cached right ?

Not sure. I don't think it will have any affect. Zend shouldn't know about the rewrite. The request is translated by Apache before it gets to the PHP engine or the Zend stuff.

I'm just guessing though, I don't really know.

[eva2000]

Originally posted by fastforward

I don't trust the browser method.

When I do that on threads I get 81K for the short ones and not available for on the originals. And for the forumhome I get 56K with the short url and 58K with the long url.

okay tested it on my sister's pc and dialup connection and yup it's compressed the page shows 129kb but loads in 5 - 10 secs on 56k modem

[fastforward]

Just to be double sure I enabled gzip logging in my Apache logs. You'll be pleased to know that it all works and everything is compressed as usual All the exclude and include filters work aswell, based on the original urls).

[fastforward]

Someone alerted me to a Netscape problem when using this hack.

I seems that Netscape ignores the < BASE > tag.

Subsequently, any urls that do not start with a slash or a fully qualified domain will fail.

The solution is to go through all the showthread, forumhome and forumdisplay templates and prefix all relative HREF's with the $ bburl variable.

This means the hack isn't quite so 'quick and easy' after all

If someone knows why Netscape is doing this please let us know.

[eva2000]

Originally posted by fastforward
Someone alerted me to a Netscape problem when using this hack.

I seems that Netscape ignores the < BASE > tag.

Subsequently, any urls that do not start with a slash or a fully qualified domain will fail.

The solution is to go through all the showthread, forumhome and forumdisplay templates and prefix all relative HREF's with the $ bburl variable.

This means the hack isn't quite so 'quick and easy' after all

If someone knows why Netscape is doing this please let us know.

really ? hmmmmm