Spider friendly URLs

For vB 2.0

This little hackette is a quick fix to allow search engine bots to spider your threads.

Although this will allow the bots to index every thread on your site, it will not make the threads 'search engine optimized'. They will see exactly what you see when you visit your site. It simply removes the CGI bits from the URL's which prevents most search engine bots from spidering more than one level deep.

If you want a hack that allows to fully customize how the thread will look to the search engine bot, you should look at Overgrows more complete hack here.

The advantage of this hack over Overgrows is that it does not require htaccess support which can have performance issues. This could also be seen as a disadvantage though as my hack requires that you have mod_rewrite enabled on your Apache Server, whereas Overgrows method should work with just about any web host out there.

Take yer pick

[fastforward]

Are you on a dedicated server? If not, you may not have access httpd.conf. If you are on a dedicated, what is the OS and what is the web server or who is your host?

I believe somebody installed this using .htaccess although I have not done so.

[Velocd]

I'm not on a dedicated server..so that is probably the reason I cannot find the file.

[high]* Velocd snaps his fingers then walks away[/high]

[jOOPeON]

Hi guys,

I installed this great hack couple weeks ago and everything went like charm.
Thread links looks just like they should look ( domain.com/t27927.html),
forums looks ok (domain.com/f232)....

I've been watching Google to visit all the forums, but for some reason it doesn't touch the threads?
Have they changed the bot or what could be wrong?
Anyway the googlebots visits the site daily and every day it spiders everything else but threads??

-jOOP

[jOOPeON]

Originally posted by jOOPeON
Hi guys,

I installed this great hack couple weeks ago and everything went like charm.
Thread links looks just like they should look ( domain.com/t27927.html),
forums looks ok (domain.com/f232)....

I've been watching Google to visit all the forums, but for some reason bots doesn't touch the threads?
Have they changed the bot or what could be wrong?
Anyway the googlebots visits the site daily and every day they spiders everything else but threads??

-jOOP

Now i changed also the forum links to look like www.domain.com/f1231.html

Will see what happens?

-jOOP

[Paul]

[QUOTE]Originally posted by eva2000
got it to work now with this

RewriteEngine on
RewriteRule ^/f([0-9]+)/?$ /forumdisplay.php?forumid=$1 [L]
RewriteRule ^/f([0-9]+)/s?$ /forumdisplay.php?forumid=$1 [L]
RewriteRule ^/t([0-9]+)\.html$ /showthread.php?threadid=$1 [L]
RewriteRule ^/t([0-9]+)/s([^/]?)\.html$ /showthread.php?threadid=$1&s=$2 [L]
RewriteRule ^/s([^/\?]0-9)+/$ /index.php?s=$1 [L]

[Paul]

[QUOTE]Originally posted by fastforward

easypeasy
In online.php : Change the 'default' block of the CASE statement on or around line 626 to look like this:

  default:
    if (preg_match ("/t([0-9]+)\.html/", $filename, $match)) {
       $userinfo[threadid] = $match[1];
       $userinfo[activity] = 'showthread';
       $threadids .= ",$userinfo[threadid]";
    } elseif (preg_match ("/f([0-9]+)/", $filename, $match)) {
       $forumid = $match[1];
       $forumids .= ",$forumid";
       $userinfo[activity] = 'forumdisplay';
       $userinfo[forumid] = $forumid;
    } elseif (preg_match ("/robots\.txt/", $filename)) {
       $userinfo[activity] = 'spider';
    } else {
      $userinfo[activity] = 'unknown';
    }
  }

[Paul]

Now--this should be interesting. I'm having quite a serious issue with my forum, and I'm guessing it has something to do with this hack. Prior to installing it, we hadn't seen it before.

I'm using the following rewrite rules:

        RewriteEngine on
        RewriteRule ^/forumdisplay.php(.*)$ /forums/forumdisplay.php$1 [R=permanent,L]
        RewriteRule ^/f([0-9]+)/?$ /forums/forumdisplay.php?forumid=$1 [L]
        RewriteRule ^/f([0-9]+)/s([^/]+?)\.html$ /forums/forumdisplay.php?forumid=$1&s=$2 [L]
        RewriteRule ^/f([0-9]+)/t([0-9]+)/?$ /forums/showthread.php?threadid=$2 [L]
        RewriteRule ^/f([0-9]+)/t([0-9]+)/s([^/]+?)\.html$ /forums/showthread.php?threadid=$2&s=$3 [L]
        RewriteRule ^/t([0-9]+)\.html$ /forums/showthread.php?threadid=$1 [L]
        RewriteRule ^/t([0-9]+)/s([^/]+?)\.html$ /forums/showthread.php?threadid=$1&s=$2 [L]
        RewriteRule ^/s([^/]+?)\.html$ /forums/index.php?s=$1 [L]

        RewriteRule ^/forums/f([0-9]+)/?$ /forums/forumdisplay.php?forumid=$1 [L]
        RewriteRule ^/forums/f([0-9]+)/s([^/]+?)\.html$ /forums/forumdisplay.php?forumid=$1&s=$2 [L]
        RewriteRule ^/forums/f([0-9]+)/t([0-9]+)/?$ /forums/showthread.php?threadid=$2 [L]
        RewriteRule ^/forums/f([0-9]+)/t([0-9]+)/s([^/]+?)\.html$ /forums/showthread.php?threadid=$2&s=$3 [L]
        RewriteRule ^/forums/t([0-9]+)\.html$ /forums/showthread.php?threadid=$1 [L]
        RewriteRule ^/forums/t([0-9]+)/s([^/]+?)\.html$ /forums/showthread.php?threadid=$1&s=$2 [L]
        RewriteRule ^/forums/s([^/]+?)\.html$ /forums/index.php?s=$1 [L]

I did it this way so that www.domain.com/f1 links would work as well as www.domain.com/forums/f1--so far so good. The problem is that every time I follow a link, a new session hash is generated--this means all sorts of login problems.

Could this be related to this hack in some way? I don't even know where to start!

Thanks,
Paul

Edit: I also made an adjustment to online.php. My version looks like this:

PHP Code:

  // Hack: Allow WOL to work with spider friendly URLs - Nov 2, 2002 - Paul
  /*
    $userinfo[activity] = 'unknown';
  */
    if (preg_match ("/t([0-9]+)/", $loc, $match)) {
       $userinfo[threadid] = $match[1];
       $userinfo[activity] = 'showthread';
       $threadids .= ",$userinfo[threadid]";
    } elseif (preg_match ("/f([0-9]+)/", $loc, $match)) {
       $forumid = $match[1];
       $forumids .= ",$forumid";
       $userinfo[activity] = 'forumdisplay';
       $userinfo[forumid] = $forumid;
    } elseif (preg_match ("/robots\.txt/", $filename)) {
       $userinfo[activity] = 'spider';
    } else {
      $userinfo[activity] = 'unknown';
    }
  // End hack: Allow WOL to work with spider friendly URLs 


[Erwin]

Change &s=$2 to just &s= in your .htaccess and see what happens. It works for me. However, this way, sessionhash is always cleared. I have had no problems with any of my users - I just tell them they need to enable cookies for my site to work.

[Paul]

Hi Erwin,

Thanks for the suggestion. The *REALLY* weird part about this is that I'm using the vbulletin URL's to test this out--it DOESN'T happen when I use the rewrite urls. (i.e. http://www.domain.com/forums/index.php?s=######...)

I don't want to force them to use cookies though ... I'm staring at the apache rewrite rule manual now trying to figure out what I'm missing here.

[Paul]

I removed the redirects completely, restarted apache, and uploaded the unhacked online.php--the problems continued to persist. I wonder if this might be a real vb bug