Spider friendly URLs

For vB 2.0

This little hackette is a quick fix to allow search engine bots to spider your threads.

Although this will allow the bots to index every thread on your site, it will not make the threads 'search engine optimized'. They will see exactly what you see when you visit your site. It simply removes the CGI bits from the URL's which prevents most search engine bots from spidering more than one level deep.

If you want a hack that allows to fully customize how the thread will look to the search engine bot, you should look at Overgrows more complete hack here.

The advantage of this hack over Overgrows is that it does not require htaccess support which can have performance issues. This could also be seen as a disadvantage though as my hack requires that you have mod_rewrite enabled on your Apache Server, whereas Overgrows method should work with just about any web host out there.

Take yer pick

[jon2x]

<waffle>
just a thought....
by replacing:

<a href="showthread.php?s=$session[sessionhash]&threadid=$thread[threadid]">

with

<a href="$bburl/t$thread[threadid].html">

i believe you're preventing the sessionhash from being moved direct between pages?... does this affect users who aren't using cookies to track their sessionhash?

if two users with the same browser behind the same proxy and without cookies enabled try to access the site they might get strange things happening like posts being marked as read that they haven't seen etc....

if this is the case i don't think there's a way around it because passing the sessionhash in the path_info might cause spiders from different ips to register multiple copies of the same page...

sorry if i'm missing the point of sessionhash or the way this hack works aranoid:
</waffle>

[The Piper]

By reading your post I got kinda worried about this as well. But I just checked the "Who's Online" page on my site, and all of the pages are something like:

http://www.pinkfloydfan.net/t145.html?s=SESSIONHASH

So, apparently, the sessionhash is being passed through the new search-engine friendly page.

BTW, I just tried to update my online.php file with the instructions I found earlier (and posted on the compilation file before), but it didn't work. I was supposed to find this:

$loc=preg_replace("/?s=[a-z0-9]{32}(&)?/","?",$loc);

But only found this, on 2.2.6:

$loc=preg_replace("/\?s=[a-z0-9]{32}(&)?/","?",$loc);

I've tried adding

$loc=preg_replace("/(f)([0-9]+)(/)/","forumdisplay.php?forumid=\2",$loc);
$loc=preg_replace("/(t)([0-9]+)(.html)/","showthread.php?threadid=\2",$loc);
$loc=preg_replace("/&s=[a-z0-9]+/","",$loc);

but then I got lots of errors. No idea of what is going on....

[jon2x]

when i go to your site (http://www.pinkfloydfan.net/) none of the links on the frontpage are actually in the "spider friendly" format... how are the searchengine spiders working?

[The Piper]

Actually, some of them are in spider-friendly version. Some of them are not, YET, because of a few things: First, I want some of the forums to be shown ordered by Thread in ascendant order, so that can't be accomplished yet with spider-friendly URLS. Second, a few of the regular links (such as those "Comments" on every news) don't need to be SE optimized, because I have the link to the news forum in a SE friendly format, on the left menu.

[eva2000]

hi long time no see

i installed the hack and working fine but moving servers from RH 6.2 apache 1.3.26 to RH 7.3 1.3.26 i'm getting an error when i restart apache

i have in my virtualHost directive

RewriteEngine on
RewriteRule ^/f([0-9]+)/s([^/]+?)$ /forumdisplay.php?forumid=$1&s=$2 [L]
RewriteRule ^/t([0-9]+)/s([^/]+?)\.html$ /showthread.php?threadid=$1&s=$2 [L]
RewriteRule ^/s([^/])+?/$ /index.php?s=$1 [L]

and get this error in restarting apache ?

Syntax error on line 1688 of /usr/local/apache/conf/httpd.conf:
RewriteRule: cannot compile regular expression '^/f([0-9]+)/s([^/]+?)$'

my current forum urls are set as

http://animeboards.com/f1/s
http://animeboards.com/t42538/s.html

hoping you can help me

[eva2000]

got it to work now with this

RewriteEngine on
RewriteRule ^/f([0-9]+)/?$ /forumdisplay.php?forumid=$1 [L]
RewriteRule ^/f([0-9]+)/s?$ /forumdisplay.php?forumid=$1 [L]
RewriteRule ^/t([0-9]+)\.html$ /showthread.php?threadid=$1 [L]
RewriteRule ^/t([0-9]+)/s([^/]?)\.html$ /showthread.php?threadid=$1&s=$2 [L]
RewriteRule ^/s([^/\?]0-9)+/$ /index.php?s=$1 [L]

[SpiceGirl]

ok, one final time please? can someone post the EXACT steps? of course, only putting some code in the .htaccess (the rewriteEngine stuff) wont do jack. we need to edit our .php files as well, right?

so which ones to edit? where? how?

thanks in advance!

[fastforward]

Originally posted by SpiceGirl
ok, one final time please? can someone post the EXACT steps? of course, only putting some code in the .htaccess (the rewriteEngine stuff) wont do jack. we need to edit our .php files as well, right?

so which ones to edit? where? how?

thanks in advance!

Here's the steps:

1. Edit httpd.conf and use Eva2000's latest rules: ie.
   

   RewriteEngine on
   RewriteRule ^/f([0-9]+)/?$ /forumdisplay.php?forumid=$1 [L]
   RewriteRule ^/f([0-9]+)/s?$ /forumdisplay.php?forumid=$1 [L]
   RewriteRule ^/t([0-9]+)\.html$ /showthread.php?threadid=$1 [L]
   RewriteRule ^/t([0-9]+)/s([^/]?)\.html$ /showthread.php?threadid=$1&s=$2 [L]
   RewriteRule ^/s([^/\?]0-9)+/$ /index.php?s=$1 [L]

2. Add the following line to your 'Head Insert' template:

   <base href="$bburl">

3. Edit the following templates:
   forumdisplay_forumbit_level1_nopost
   forumdisplay_forumbit_level1_post
   forumdisplay_forumbit_level2_nopost
   forumdisplay_forumbit_level2_post
   forumhome_forumbit_level1_nopost
   forumhome_forumbit_level1_post
   forumhome_forumbit_level2_nopost
   forumhome_forumbit_level2_post
   

   Change the links to look like this:
   <a href="$bburl/f$forum[forumid]/s$session[sessionhash]">$forum[title]</a>

4. Edit the following template:
   forumdisplaybit
   

   Change the links to look like this:
   <a href="$bburl/t$thread[threadid]/s$session[sessionhash].html">$thread[title]</a>

If you have problems with missing images in Netscape, you should go though all templates and edit the links to images to include the full path, ie' $bburl/{images_folder}/a.gif. You can use the template search function for this.

This will spider the first page of all threads only. In most cases, this should be more than enough. It is possible to add support for multipage threads, but this requires editing the code and it will probably change from version to version of vB. If you really need this, there are posts in this thread that discuss it.

I personally use the basic method. It's given me 71,800 entries in Googe for dbforums.com and 47,500 for britishexpats.com.

[tpearl5]

The only problem I'm having is with online.php displaying unknown locations. I tried the code posted here, but got errors. I'm running 2.2.5. Anyone ?]

Take a look: http://degster.com/midi/board/

[nuno]

Paul, how do we fix the Unknown Location Fields showing up in WOL page?
TIA