Spider friendly URLs

For vB 2.0

This little hackette is a quick fix to allow search engine bots to spider your threads.

Although this will allow the bots to index every thread on your site, it will not make the threads 'search engine optimized'. They will see exactly what you see when you visit your site. It simply removes the CGI bits from the URL's which prevents most search engine bots from spidering more than one level deep.

If you want a hack that allows to fully customize how the thread will look to the search engine bot, you should look at Overgrows more complete hack here.

The advantage of this hack over Overgrows is that it does not require htaccess support which can have performance issues. This could also be seen as a disadvantage though as my hack requires that you have mod_rewrite enabled on your Apache Server, whereas Overgrows method should work with just about any web host out there.

Take yer pick

[amk35]

Lierduh,

Thanks for the response

At
http://domain.com/forums/f1.html

The links on the bottom to view older threads are as follows:

2 - http://domain.com/forums/forumdispla...page=40-2.html

3 - http://domain.com/forums/forumdispla...page=40-3.html

etc. All bring up the same initial forum page when clicked

My rewrite rules are:

RewriteEngine on
RewriteRule ^/forums/f([0-9]+)/?$ /forums/forumdisplay.php?forumid=$1 [L]
RewriteRule ^/forums/f([0-9]+)\.html$ /forums/forumdisplay.php?forumid=$1 [L]
RewriteRule ^/forums/f([0-9]+)-([0-9]+)\.html$ /forums/forumdisplay.php?forumid=$1&daysprune=1000&sortord er=&sortfield=lastpost&perpage=25&pagenumber=$2 [L]
RewriteRule ^/forums/t([0-9]+)\.html$ /forums/showthread.php?threadid=$1 [L]
RewriteRule ^/forums/t([0-9]+)-([0-9]+)-([0-9]+)\.html$ /forums/showthread.php?threadid=$1&perpage=$2&pagenumber=$ 3 [L]
RewriteRule ^/forums/t([0-9]+)-([0-9]+)--(.*)--([0-9]+)\.html$ /forums/showthread.php?threadid=$1&perpage=$2&highlight=$3 &pagenumber=$4 [L]

RewriteCond %{QUERY_STRING} ^(.*)-([0-9]+)\.html$
RewriteRule ^/forums/memberlist.php$ /forums/memberlist.php?%1&pagenumber=%2? [L]
RewriteCond %{QUERY_STRING} ^(.*)-([0-9]+)\.html$
RewriteRule ^/forums/search.php$ /forums/search.php?%1&pagenumber=%2? [L]



Thanks for any help. BTW, I have no problem paying you for fixing it either I just need to get it working quickly. Thanks.

Allen

[lierduh]

Looks like you have not edited the php file.

==== >8
open forumdisplay.php

Find
$pagenav =
getpagenav($totalthreads,"forumdisplay.php?s=$sess ion[sessionhash]&forumid=$forumid&daysprune=$daysprune&sortorder=$ sortorder&sortfield=$sortfield&perpage=$perpage");

replace it with:
$pagenav = getpagenav($totalthreads,"f$forumid");
====== >8

You need to find the php file (forumdisplay.php) and edit it according to my original post. You can put "//" without the quote in front of that line to comment out the old code and add the line of new code next line to it.

I will try to come back to this forum soon to see if you have fixed it. Thanks but no, I will not accept payment for something like this.

[tpearl5]

I changed servers again and unfortunately I can't get this hack working. The server has ensim on the backend running redhat. In the .htaccess I have:

PHP Code:

RewriteEngine on 
RewriteRule ^f([0-9]+)/?$ /forumdisplay.php?forumid=$1&s=$2 [L] 
RewriteRule ^t([0-9]+).html$ /showthread.php?threadid=$1 [L]
RewriteRule ^t([0-9]+)-([0-9]+).html$ /showthread.php?threadid=$1&perpage=15&pagenumber=$2 [L]
RewriteRule ^f([0-9]+)-([0-9]+)/$ /forumdisplay.php?forumid=$1&daysprune=1000&sortorder=&sortfield=lastpost&perpage=40&pagenumber=$2 [L]
RewriteRule ^[ft]([0-9]+)/s([^/]*)/(.+)$ /$3 [L] 


However this causes a forbidden message on all directory pages. It does not however effect http://board.midibuddy.net .. (the page still loads)

The board pages are at http://board.midibuddy.net

Please help! What could I possibly be doing wrong?

[amk35]

lierduh,

Thanks very much. such a stupid oversight....works great now.

[MarkG]

i've moved to a new server and this code doesnt work anymore for some reason

if it you can get it working either here's another way:

1) set a 404.php as your 404 handler in your .htacess or httpd.conf

ErrorDocument 404 /404.php

and put this in

<?
ob_start();
if ($ref=="") $ref=$REQUEST_URI;
if (preg_match ("#/forums/t\d*-s.html#",$ref)) {
        $ref=str_replace("/forums/t","",$ref);
        $ref=str_replace("-s.html","",$ref);
        Header("Location: http://domain.com/forums/showthread.php?threadid=".$ref );
}
if (preg_match ("#/forums/f\d*-s.html#",$ref)) {
        $ref=str_replace("/forums/f","",$ref); 
        $ref=str_replace("-s.html","",$ref);
        Header("Location: http://domain.com/forums/forumdisplay.php?forumid=".$ref );      
}
ob_end_clean();
?>

important: my forums are under /forums and i have my links like t4444-s.html not t4444/s555555.html (with a - not / and without the session number). change the code according to your situation

if anyone uses it, let me know if it works for you

ps. there one issue with this sollution: you still get a line in your error log about a missing file...

[Slynderdale]

Is there a .htaccess version of this? I tried the rewrite rule in the first post and got a 500 internal error.

[MrNase]

I don't get behind that mod_rewrite stuff

Could someone try to explaine what i should add to my .htaccess when i want to redirect:

http://www.mydomain.com/pages/index?firma=john borne
or
http://www.mydomain.com/pages/index?firma=iwonttellyou

to

http://www.mydomain.com/pages/john borne
or
http://www.mydomain.com/pages/iwonttellyou

??

thx in advance

[buro9]

I'm using Google AdSense to pay for my forum, and the side effect of having the ads on the site is that they send thousands of spiders to try and figure out what is going on and what contextual adverts to place.

So... to this thread I came.

Big thanks to fastforward and lierduh for all their work.

But... I have found a couple of small bugs... most of them I've ironed out (unfortunately they were painful to find so it took a while and I can't recall every little thing I've done)... but there is one that is lingering:

In forumdisplay.php, when I'm viewing a forum:
http://www.bowlie.com/forum/f2.html

If I sort the forum by the number of thread replies. It's fine.
But then if I page onto the second page of the results... it's forgotten that I wanted it sorted.

A user pointed this out to me... but in essence... forum sorting doesn't work under this system.

I'll try and figure it out myself, but I feel I'm not as versed in mod-rewrite and the subtleties as others here... so anyone who can help, please do

[buro9]

OK, got it working again...

Here's the modification I made:

In forumdisplay.php lierduh changed:

PHP Code:

$pagenav = getpagenav($totalthreads,"forumdisplay.php?s=$session[sessionhash]&forumid=$forumid&daysprune=$daysprune&sortorder=$sortorder&sortfield=$sortfield&perpage=$perpage 


To:

PHP Code:

$pagenav = getpagenav($totalthreads,"f$forumid"); 


And I've now changed it to:

PHP Code:

$pagenav = getpagenav($totalthreads,"f$forumid--$daysprune--$sortfield--$sortorder"); 


Now... none of that is going to work without the rewrite rules... so I've added two more... but here are all the ones I have that relate to the forumdisplay.php file. Note that the first one is lierduh's original one:

  #
  # forumdisplay.php rewriting
  #
  # f2.html = forumid
RewriteRule ^f([0-9]+)\.html$ forumdisplay.php?forumid=$1 [L]
  #
  # f2--25--lastpost---2.html = forumid + daysprune + sortfield + sortorder + pagenumber
RewriteRule ^f([0-9]+)--([0-9]+)--([a-z]+)---([0-9]+)\.html$ forumdisplay.php?forumid=$1&daysprune=$2&sortorder=&sortfield=$3&perpage=25&pagenumber=$4 [L]
  #
  # f2--25--lastpost--asc-2.html = forumid + daysprune + sortfield + sortorder + pagenumber
RewriteRule ^f([0-9]+)--([0-9]+)--([a-z]+)--([a-z]+)-([0-9]+)\.html$ forumdisplay.php?forumid=$1&daysprune=$2&sortorder=$4&sortfield=$3&perpage=25&pagenumber=$5 [L]

Now all of the ordering stuff works on the page spanning, and the page spanning is easy to spider

[buro9]

Just checked, and lierduh you also suffer from this bug... so I'll assume that everyone who has done this also suffers from it.

The change I've made isn't aesthetically pleasing, but it does work and stays true to the spider friendly purpose.

To summarise to those joining the end of this thread and wondering where to begin:

First, follow fastforward's instructions on the first page of this thread... you need to do this before anything else to be familiar with what is going on before it gets scarier:

https://vborg.vbsupport.ru/showthrea...threadid=18035

Then follow lierduh's instructions within his post... there are several major improvements over the initial code.

https://vborg.vbsupport.ru/showthrea...137#post348137

If you're confident at this point and everything is working, then go through the whole codebase and replacing the relevant forumdisplay and showthread parts with the new format... stripping out sessionhashes as you go along (I've disabled the option in user prefs to not use cookies and made cookies compulsory).

Finally... to resolve the nagging bug about sorting and page spanning, do the stuff in my post above:

https://vborg.vbsupport.ru/showthrea...256#post418256

It's a lot of work, and certainly daunting. Would I attempt it if I were seriously unsure about this stuff? Nope. But if you are comfortable working with the template system and editing the php files... then the benefits of being spidered are considerable and can do a lot for your forum (increase members or increase advert revenue).