Spider friendly URLs

For vB 2.0

This little hackette is a quick fix to allow search engine bots to spider your threads.

Although this will allow the bots to index every thread on your site, it will not make the threads 'search engine optimized'. They will see exactly what you see when you visit your site. It simply removes the CGI bits from the URL's which prevents most search engine bots from spidering more than one level deep.

If you want a hack that allows to fully customize how the thread will look to the search engine bot, you should look at Overgrows more complete hack here.

The advantage of this hack over Overgrows is that it does not require htaccess support which can have performance issues. This could also be seen as a disadvantage though as my hack requires that you have mod_rewrite enabled on your Apache Server, whereas Overgrows method should work with just about any web host out there.

Take yer pick

[cyc]

Originally posted by lierduh


No worries, mate. (some downunder talking there)
Glad it is working for you.

Don't forget the last hack (RewriteCond) regarding the memberlist. Otherwise your member list navigation will not work. Fortunately this is a simple one, only need to add two lines to your rewriterules.

Waiting for more things to break, I am sure there is. eg. Merge Thread does not work anymore due to the new URL. Do I care to fix this is another question.

Yep done that now thanks for the tip!

I'm thinking I might as well kill my archive now :smoke:

cheers mate!

[Haakon]

Hi lierduh, sorry to bother you about this. But I need some help concerning the approvements you`ve done to the spider friendly hack. I`ve gone through all the steps in your hack, it works great in every area except one, the forumdisplay part.
The first problem is with multipage threads, in forumdisplay it will only show a error page if I click the page buttons. I`ve disabled that hack but the url shows something like "f123--2" which is strange. Any theories about this one? What should I check?

The second problem is that the pagenumber buttons are a bit funky, test this one http://www.radiomute.com/f112-3.html. Click on different pages, unexplainable

I`m completely aware that this is not your problem, as I can see you`re hack works perfect for everyone else, but I don`t understand this though I`ve looked into it.

Cheers,
Haakon

[lierduh]

1.

I presume you mean the multiple page thread links displayed under a forumdisplay.

It should be something like t3593-15-5.html

If you had it as: f123--2
then your template is not done properly. The template you need to change is:

================
For forumdisplay_multipagenav_pagenumber

Replace
<a href="showthread.php?s=$session[sessionhash]&threadid=$thread[threadid]&perpage=$pperpage&pagenumber=$acurpage">$acurpage </a>

with
<a href="t$thread[threadid]-$pperpage-$acurpage.html">$acurpage</a>
======================

2.

My original rewriterule set the "daysprune=30" (the second rewriterule). Your default setting for "Default view age" is "Last Day". You need to make them the same.

My own forums setting at the moment is "the beginning" for the "Default view age" and the rewriterule is "daysprune=1000"
That will show all the threads, making them all ready to feed the Googlebots.

You can set "Default view age" for each forum through Admin panel, click Modify forums and then "setting". As the side note says, 1000 means from the beginning.

[Haakon]

Hi, changed the rewrite rule to my settings and it works with no funky behavior..... thanks!

The 1st problem didn`t get solved though. It`s like it dosn`t recognize the $pperpage variable. The pages will get spidered from the showthread templates though, so it`s no krisis.

Thanks again,
Haakon

[cobradude]

Is there a way to NOT monkey with the links for multiple pages for memberlist or search? I block the bots for those anyway, and the more rules I have, the easier it will be to break down the road. Any solutions?

[lierduh]

Originally posted by cobradude
Is there a way to NOT monkey with the links for multiple pages for memberlist or search? I block the bots for those anyway, and the more rules I have, the easier it will be to break down the road. Any solutions?

Unfortunately the navigation link templates are used for all the multiple pages. So the additional rules for memberlist and search are necessary. Fortunately they are only rules to add without the need to muck around with templates or scripts. I think the additional rules are pretty safe.

If you do not want the robots to index your memberlist or search results (I don't think they will go and fill your search terms anyway). Just create a robots.txt file and they won't bother.

[The Piper]

First, I'd like to thank lierduh for reworking the original hack. It's finally working like a charm on my site.

I'm having one bit of a problem, though. I've noticed, by looking at Who's Online, that some users have unknown locations such as those:

/t2094-3.html ? s = xxxxxxxxxxxxxxxxxxx
/f8 ? s = xxxxxxxxxxxxxxxxxxx
/f13 ? s = xxxxxxxxxxxxxxxxxxx

Note: Extra spaces added, because vB.org removes the session part, when posted as it is. xxxxxx in place of session as well to avoid problems.

I was using the original version of the hack, so I used to have URLs like http://www.pinkfloydfan.net/f8, instead of http://www.pinkfloydfan.net/f8.html, so the second and third unknown locations might be because of old references that users posted. (Any clue on how to update old references in posts?), but I have no clue of why the first location (/t2094-3.html ? s = xxxxx) is showing up like that.

Any ideas on this? Thanks a lot again!

[lierduh]

Not sure why the session id is added.

Perhaps you could check the web log, see extractly what URL was accessed, ie. with or without the session id. It may also help you to find out where the link was originated, ie. know as the referrer.

[007]

This is probably a stupid question... but where would I find httpd.conf??? I can't find it in anywhere...

[dstruct2k]

httpd.conf is part of the Apache installation... this mod assumes that you're running your own server....

Too bad it isn't working for me anyways. (Latest version of Apache)

[The Piper]

I'm having some issues with forumdisplay. Please check http://www.pinkfloydfan.net/f20.html and look at the page navigation. Click on any of them, say, page 2. I'm taken to an URL such as http://www.pinkfloydfan.net/forumdis...age=50-2.html, which does not work (that is, I'm shown the first page of the forum as well).

I'd really appreciate any input on this, please. I've changed everything according to this post, and have already doublechecked it.

Thanks a lot,
Paulo

[The Piper]

Ok, since the issues I brought up on my last post were also happening in other parts of the site (for instance, in the Control Panel, while choosing an avatar, users could not choose any page other than 2 - we've got lots of avatars), he would get an URL such as the one described on the past post.

I've just uninstalled all the parts of this hack that deal with page navigation and, of course, everything is working fine again. Of course search engines won't be indexing any pages other than page 1 of all of my threads, but at least everything is working.

If anyone got these problems and/or wants to try to fix them, I'll try to help.

Just thought I'd post my experiences here.

[slinky]

There has to be an easy way of doing this... all my pages are html using the mod rewrite and I've made the mod a long, long time ago....

[PurpleCow]

Hii all,

This is a TOP most important hack for vBulletin that was ever done and will be ever done ( might be ), if someone can get it to work with every aspect/page of vbulletin (incase it is yet to be done ?? ) !

I would want to use this hack on my yet to be launched forums and i'd like to know, if the attachment in the first post of this thread is a workiable version for this hack ?? If not kindly attach a file with instructions to get it to work !

I do have a suggestion for fastforward and all other lead developers, contributing developers and behind the scene developers :

Kindly contact Brett_tbake of WebMasterWorld, who has successfully implemented this kind of hack for his forums. however, we should keep in mind that his forums are custom coded and not using vBulletin. but what i mean to say is that, he can sure give you some inputs to make this hack easy to use and get it to work.

I request one of you developers get in touch with him there at webmasterworld.com.

I do have another question :

My forums are installed in a subdomain, meaning.... http://xyz.mydomain.com and not /forums/ type of directory, if i have to get this to work, what if any changes should i make ??

Also, kindly some one give me a link to the thread for Overgorw's hack on this , none of the previously potsed url's work for me !

As a sidenote, i'd like to mention that i am using version 2.3.0, and i hope this hack works for this latest version ??

Thank you very much for working on this hack.

Cheers

[slinky]

My hack seems to work. It's simple but I have no idea if they are being properly indexed. Search engines change their search methods over time and it is tough to track. I've been keeping my forums as is (my pages end in .html files) since it works and little hassle.

But hacks are hacks. A writer is under no obligation to provide support whatsoever so we have to live with it.