Spider friendly URLs

For vB 2.0

This little hackette is a quick fix to allow search engine bots to spider your threads.

Although this will allow the bots to index every thread on your site, it will not make the threads 'search engine optimized'. They will see exactly what you see when you visit your site. It simply removes the CGI bits from the URL's which prevents most search engine bots from spidering more than one level deep.

If you want a hack that allows to fully customize how the thread will look to the search engine bot, you should look at Overgrows more complete hack here.

The advantage of this hack over Overgrows is that it does not require htaccess support which can have performance issues. This could also be seen as a disadvantage though as my hack requires that you have mod_rewrite enabled on your Apache Server, whereas Overgrows method should work with just about any web host out there.

Take yer pick

[cyc]

Originally posted by lierduh


No worries, mate. (some downunder talking there)
Glad it is working for you.

Don't forget the last hack (RewriteCond) regarding the memberlist. Otherwise your member list navigation will not work. Fortunately this is a simple one, only need to add two lines to your rewriterules.

Waiting for more things to break, I am sure there is. eg. Merge Thread does not work anymore due to the new URL. Do I care to fix this is another question.

Yep done that now thanks for the tip!

I'm thinking I might as well kill my archive now :smoke:

cheers mate!

[Haakon]

Hi lierduh, sorry to bother you about this. But I need some help concerning the approvements you`ve done to the spider friendly hack. I`ve gone through all the steps in your hack, it works great in every area except one, the forumdisplay part.
The first problem is with multipage threads, in forumdisplay it will only show a error page if I click the page buttons. I`ve disabled that hack but the url shows something like "f123--2" which is strange. Any theories about this one? What should I check?

The second problem is that the pagenumber buttons are a bit funky, test this one http://www.radiomute.com/f112-3.html. Click on different pages, unexplainable

I`m completely aware that this is not your problem, as I can see you`re hack works perfect for everyone else, but I don`t understand this though I`ve looked into it.

Cheers,
Haakon

[lierduh]

1.

I presume you mean the multiple page thread links displayed under a forumdisplay.

It should be something like t3593-15-5.html

If you had it as: f123--2
then your template is not done properly. The template you need to change is:

================
For forumdisplay_multipagenav_pagenumber

Replace
<a href="showthread.php?s=$session[sessionhash]&threadid=$thread[threadid]&perpage=$pperpage&pagenumber=$acurpage">$acurpage </a>

with
<a href="t$thread[threadid]-$pperpage-$acurpage.html">$acurpage</a>
======================

2.

My original rewriterule set the "daysprune=30" (the second rewriterule). Your default setting for "Default view age" is "Last Day". You need to make them the same.

My own forums setting at the moment is "the beginning" for the "Default view age" and the rewriterule is "daysprune=1000"
That will show all the threads, making them all ready to feed the Googlebots.

You can set "Default view age" for each forum through Admin panel, click Modify forums and then "setting". As the side note says, 1000 means from the beginning.

[Haakon]

Hi, changed the rewrite rule to my settings and it works with no funky behavior..... thanks!

The 1st problem didn`t get solved though. It`s like it dosn`t recognize the $pperpage variable. The pages will get spidered from the showthread templates though, so it`s no krisis.

Thanks again,
Haakon

[cobradude]

Is there a way to NOT monkey with the links for multiple pages for memberlist or search? I block the bots for those anyway, and the more rules I have, the easier it will be to break down the road. Any solutions?

[lierduh]

Originally posted by cobradude
Is there a way to NOT monkey with the links for multiple pages for memberlist or search? I block the bots for those anyway, and the more rules I have, the easier it will be to break down the road. Any solutions?

Unfortunately the navigation link templates are used for all the multiple pages. So the additional rules for memberlist and search are necessary. Fortunately they are only rules to add without the need to muck around with templates or scripts. I think the additional rules are pretty safe.

If you do not want the robots to index your memberlist or search results (I don't think they will go and fill your search terms anyway). Just create a robots.txt file and they won't bother.

[The Piper]

First, I'd like to thank lierduh for reworking the original hack. It's finally working like a charm on my site.

I'm having one bit of a problem, though. I've noticed, by looking at Who's Online, that some users have unknown locations such as those:

/t2094-3.html ? s = xxxxxxxxxxxxxxxxxxx
/f8 ? s = xxxxxxxxxxxxxxxxxxx
/f13 ? s = xxxxxxxxxxxxxxxxxxx

Note: Extra spaces added, because vB.org removes the session part, when posted as it is. xxxxxx in place of session as well to avoid problems.

I was using the original version of the hack, so I used to have URLs like http://www.pinkfloydfan.net/f8, instead of http://www.pinkfloydfan.net/f8.html, so the second and third unknown locations might be because of old references that users posted. (Any clue on how to update old references in posts?), but I have no clue of why the first location (/t2094-3.html ? s = xxxxx) is showing up like that.

Any ideas on this? Thanks a lot again!

[lierduh]

Not sure why the session id is added.

Perhaps you could check the web log, see extractly what URL was accessed, ie. with or without the session id. It may also help you to find out where the link was originated, ie. know as the referrer.

[007]

This is probably a stupid question... but where would I find httpd.conf??? I can't find it in anywhere...

[dstruct2k]

httpd.conf is part of the Apache installation... this mod assumes that you're running your own server....

Too bad it isn't working for me anyways. (Latest version of Apache)