Spider friendly URLs

For vB 2.0

This little hackette is a quick fix to allow search engine bots to spider your threads.

Although this will allow the bots to index every thread on your site, it will not make the threads 'search engine optimized'. They will see exactly what you see when you visit your site. It simply removes the CGI bits from the URL's which prevents most search engine bots from spidering more than one level deep.

If you want a hack that allows to fully customize how the thread will look to the search engine bot, you should look at Overgrows more complete hack here.

The advantage of this hack over Overgrows is that it does not require htaccess support which can have performance issues. This could also be seen as a disadvantage though as my hack requires that you have mod_rewrite enabled on your Apache Server, whereas Overgrows method should work with just about any web host out there.

Take yer pick

[cobradude]

Originally posted by cyc
Has anyone set this up so guests get the rewritten urls and members get the normal urls? is this even possible?

cheers!

Assign a default style with these mods to your not logged in group, and assign another style without the mods to your members.

You'll need to install this hack to assign styles to usergroups: https://vborg.vbsupport.ru/showthrea...ghlight=styles

[Gutspiller]

Can somebody help me? I don't understand what I'm suppose to do in step one. Can somebody give me some more in-depth intructions on step one? What exactly am I suppose to do? Please help, I would really like to install this hack.

[lierduh]

Edit: 13 Dec 2003, posted some notes about VB version 3 Beta 7.

https://vborg.vbsupport.ru/showpost....&postcount=298

Edit: 26 July 2003, buro9 has found and fixed quite a few bugs, he has also made improvement. Please refer to his post at:

https://vborg.vbsupport.ru/showthrea...257#post418257

Edit: 9 Feb 2003, added RewriteCond to fix search result navigation problem.

Edit: 5 Feb 2003, added RewriteCond to fix member list navigation problem.

Edit: 2 Feb 2003, added hack to the forums/threads page navigation links to cover more threads. Change made to pagenumber and perpage's position from my initial post.
===========

I will share my experience with this hack.
My web server is running apache.

First of all, your web server needs to have mod_rewrite module installed and enabled.

I put rewrite rules in .htaccess (in httpd.conf should be the same)
I decided not to use session ID. I will explain this later.

First, for .htaccess to work, you need to enable it with "AllowOverride" in your httpd.conf file. My AllowOverride line looks like this:

AllowOverride AuthConfig Options FileInfo

Naturally
AllowOverride All
will work too.

For Redhat and Mandrake Linux, httpd.conf file is located under:
/etc/httpd/conf/

If you don't know where is yours, try:
#locate httpd.conf

For those of you who can not access to httpd.conf file, eg. use someone else's server etc, .htaccess is a better option.

I use .htaccess for different settings including php options for different software. You may only want rewrite rules for a certain software, not everything under the same domain!

My rewrite rules extracted from .htaccess:

RewriteEngine on
RewriteRule ^f([0-9]+)\.html$ forumdisplay.php?forumid=$1 [L]
RewriteRule ^f([0-9]+)-([0-9]+)\.html$ forumdisplay.php?forumid=$1&daysprune=30&sortorder =&sortfield=lastpost&perpage=25&pagenumber=$2 [L]
RewriteRule ^t([0-9]+)\.html$ showthread.php?threadid=$1 [L]
RewriteRule ^t([0-9]+)-([0-9]+)-([0-9]+)\.html$ showthread.php?threadid=$1&perpage=$2&pagenumber=$ 3 [L]
RewriteRule ^t([0-9]+)-([0-9]+)--(.*)--([0-9]+)\.html$ showthread.php?threadid=$1&perpage=$2&highlight=$3 &pagenumber=$4 [L]

RewriteCond %{QUERY_STRING} ^(.*)-([0-9]+)\.html$
RewriteRule ^memberlist.php$ memberlist.php?%1&pagenumber=%2? [L]
RewriteCond %{QUERY_STRING} ^(.*)-([0-9]+)\.html$
RewriteRule ^search.php$ search.php?%1&pagenumber=%2? [L]


Why don't I use session ID?

The Googlebots hate the most is session ID. As soon as they remotely detect there is an session id, they turn around and go away. This is why no vBulletin boards will be indexed. I have got a feeling that if the session id is removed, even with no rewrite hack and with the "?" in the URL, google will index the forums. The reason why Googlebots do not like session ID is: Each time the bot visits, for the same contents, it will be fed with different URL. Sometimes more than one bots might visit at the same time. They might be pulling the same page with different URL! I also believe Google punishes with lowered PR rating for those web sites that have different URLs for the same page. So even if you do rewrite session IDs as part of URL, the bots will get different URLs for the same page, when they process the data and find out, they will give your site a lowered PR rating, with passion! as your site has wasted a lot of their time.

The second reason for not using session ID is security. Imagine this, you have your cookies turned off, you go to this forums, log in and find something good, you subsequently copy the URL of one of the thread and paste in a post. As the session ID carries your Logged in ID. Now if someone else who has not even logged in click on the URL you just posted, he becomes you, he can then post under your name! Tell this to your members, then you will have no problem forcing them to turn on the cookies.

You will notice for the forums, I also use .html as the forums page instead of the original directory structure for the forums. Google lowers PR rating for each directory layer, so don't let the "/" get in the way. Also with "/", you will have image display problems etc. As the displayforums.php is in the root directly, now your page is "/f10298740/" .

The following templates need to be changed:

forumdisplay_forumbit_level1_nopost
forumdisplay_forumbit_level1_post
forumdisplay_forumbit_level2_nopost
forumdisplay_forumbit_level2_post
forumhome_forumbit_level1_nopost
forumhome_forumbit_level1_post
forumhome_forumbit_level2_nopost
forumhome_forumbit_level2_post
usercp_forumbit

Find
forumdisplay.php?s=$session[sessionhash]&forumid=$forum[forumid]

Replace it with:
f$forum[forumid].html

For forumdisplaybit template,

Replace
<a href="showthread.php?s=$session[sessionhash]&threadid=$thread[threadid]">$thread[title]</a>

with
<a href="t$thread[threadid].html">$thread[title]</a>

For forumdisplay_multipagenav_pagenumber

Replace
<a href="showthread.php?s=$session[sessionhash]&threadid=$thread[threadid]&perpage=$pperpage&pagenumber=$acurpage">$acurpage </a>

with
<a href="t$thread[threadid]-$pperpage-$acurpage.html">$acurpage</a>

To fix "Who is online" problem.

In online.php, find:

$loc=preg_replace("/\?s=[a-z0-9]{32}(&)?/","?",$loc);

add the following lines:

$loc=preg_replace("/f(\d+).html/","forumdisplay.php?forumid=\$1",$loc);
$loc=preg_replace("/t(\d+).html/","showthread.php?threadid=\$1",$loc);
$loc=preg_replace("/t(\d+)-(\d+)-(\d+).html/","showthread.php?threadid=\$1",$loc);

I will post more information when I find more problems with the hack. Hope my experience helps someone.

========= Edit: added the hack for forumdisplay.php navigation links

Fix Forums Navigation links:

open forumdisplay.php

Find
$pagenav =
getpagenav($totalthreads,"forumdisplay.php?s=$sess ion[sessionhash]&forumid=$forumid&daysprune=$daysprune&sortorder=$ sortorder&sortfield=$sortfield&perpage=$perpage");

replace it with:
$pagenav = getpagenav($totalthreads,"f$forumid");


Change the following templates as well.

pagenav_firstlink
find: <a href="$address&pagenumber=$curpage" title="first page">
change to: <a href="$address-$curpage.html" title="first page">

pagenav_lastlink
find: <a href="$address&pagenumber=$curpage" title="last page">
change to: <a href="$address-$curpage.html" title="last page">

pagenav_nextlink
find: <a href="$address&pagenumber=$nextpage" title="next page">
change to: <a href="$address-$nextpage.html" title="next page">

pagenav_pagelink
find: <a href="$address&pagenumber=$curpage">$curpage</a>
change to: <a href="$address-$curpage.html">$curpage</a>

pagenav_prevlink
find: <a href="$address&pagenumber=$prevpage" title="previous page">&laquo;</a>
change to: <a href="$address-$prevpage.html" title="previous page">&laquo;</a>

=============
Open showthread.php

find:
$pagenav = getpagenav($totalposts,"showthread.php?s=$session[sessionhash]&threadid=$threadid&perpage=$perpage".iif(isset($h ighlight), "&highlight=$highlight", ""));

change it to:

$pagenav = getpagenav($totalposts,"t$threadid-$perpage".iif(isset($highlight),"--$highlight-", ""));

[Haakon]

Thanks, especially for the multi page addon I really needed that.

Cheers,
Haakon

[cyc]

Hi,

Thats a very handy post!

I cant get the forumdisplay_multipagenav_pagenumber bit to work, i get a 404 error when going to the new short url's. Everything else has been working great for the last week or so :banana:

cheers!

[lierduh]

Originally posted by cyc
Hi,

Thats a very handy post!

I cant get the forumdisplay_multipagenav_pagenumber bit to work, i get a 404 error when going to the new short url's. Everything else has been working great for the last week or so :banana:

cheers!

Post your rewrite rules and the URL causing page 404.

[cyc]

Originally posted by lierduh

Post your rewrite rules and the URL causing page 404.

Current rewrite rules from the httpd.conf file are:

RewriteEngine on
RewriteRule ^/f([0-9]+)/?$ /forumdisplay.php?forumid=$1 [L]
RewriteRule ^/f([0-9]+)/s?$ /forumdisplay.php?forumid=$1 [L]
RewriteRule ^/t([0-9]+)\.html$ /showthread.php?threadid=$1 [L]
RewriteRule ^/t([0-9]+)/s([^/]?)\.html$ /showthread.php?threadid=$1&s=$2 [L]
RewriteRule ^/s([^/\?]0-9)+/$ /index.php?s=$1 [L]
RewriteRule ^t([0-9]+)-([0-9]+)-([0-9]+)\.html$ showthread.php?threadid=$1&perpage=$3&pagenumber=$ 2 [L]


goto http://www.cyclingforums.com/f47 and click on a number in the top thread.

cheers!

[lierduh]

Originally posted by cyc


Current rewrite rules from the httpd.conf file are:

RewriteEngine on
RewriteRule ^/f([0-9]+)/?$ /forumdisplay.php?forumid=$1 [L]

goto http://www.cyclingforums.com/f47 and click on a number in the top thread.

cheers!

Your rewrite rule is waiting for:
http://www.cyclingforums.com/f47/
not
http://www.cyclingforums.com/f47

So either you need to add a '/' to your template or just make your rewrite rule to:

RewriteRule ^/f([0-9]+)$ /forumdisplay.php?forumid=$1

Your rewrite rule is looking for:

URL matches (between "^" and "$") which is /f123/xxx
Try
^/f([0-9]+)$ or ^f([0-9]+)$

[lierduh]

The original hack does not change the forums/threads' navigation links. ie.

Pages (5): « First ... « 3 4 [5]
Pages (26): « First ... « 23 24 [25] 26 »

So basically the bots can only craw the first page of a single forum. With my addtional hack, the bots will be able to crawl all the threads specified by "Last 30 days" or whatever the default setting is.

This also allows every thread to be crawled even if they are spreaded through more than 4 (or 9) pages.

I have edited my original post to include this hack.

[Haakon]

Thanks again lierduh, can`t imagine the increase in visitors for next month

However, there is a little problem with the last hack (I think it`s the last one). If you look in member list, the page button is now disabled. Someone spotted it in my forum, and I see you`ve also got the problem. Can you fix it

I`ve also got a problem now with some of the thread`s last page will be blank. Not sure if this is because of your hack, but maybe you know something about it?

Thanks,
Haakon