Spider friendly URLs

For vB 2.0

This little hackette is a quick fix to allow search engine bots to spider your threads.

Although this will allow the bots to index every thread on your site, it will not make the threads 'search engine optimized'. They will see exactly what you see when you visit your site. It simply removes the CGI bits from the URL's which prevents most search engine bots from spidering more than one level deep.

If you want a hack that allows to fully customize how the thread will look to the search engine bot, you should look at Overgrows more complete hack here.

The advantage of this hack over Overgrows is that it does not require htaccess support which can have performance issues. This could also be seen as a disadvantage though as my hack requires that you have mod_rewrite enabled on your Apache Server, whereas Overgrows method should work with just about any web host out there.

Take yer pick

[cyc]

Originally posted by lierduh
The original hack does not change the forums/threads' navigation links. ie.

Pages (5): ? First ... ? 3 4 [5]
Pages (26): ? First ... ? 23 24 [25] 26 ?

So basically the bots can only craw the first page of a single forum. With my addtional hack, the bots will be able to crawl all the threads specified by "Last 30 days" or whatever the default setting is.

This also allows every thread to be crawled even if they are spreaded through more than 4 (or 9) pages.

I have edited my original post to include this hack.

Great job with adding to this awesome hack :banana:

I'm have an issue with the navigation for the thread lists..... http://www.cyclingforums.com/f47 (the links/numbers) would I be right in thinking its a prob with my rewrite rules?

#### edit ####

I've uninstalled the changes that relate to this post, people started complaining about dead links


thanks

[lierduh]

Edit: Also added RewriteCond for search.php to make sure multi-page search result work properly.

Originally posted by Haakon

However, there is a little problem with the last hack (I think it`s the last one). If you look in member list, the page button is now disabled. Someone spotted it in my forum, and I see you`ve also got the problem. Can you fix it

I`ve also got a problem now with some of the thread`s last page will be blank. Not sure if this is because of your hack, but maybe you know something about it?

Thanks,
Haakon

Please add the following to your existing rewriterules, it should fix the memberlist navigation problem:

RewriteCond %{QUERY_STRING} ^(.*)-([0-9]+)\.html$
RewriteRule ^memberlist.php$ memberlist.php?%1&pagenumber=%2? [L]

RewriteCond %{QUERY_STRING} ^(.*)-([0-9]+)\.html$
RewriteRule ^search.php$ search.php?%1&pagenumber=%2? [L]


Edited my original post to reflect this.

Please note, I have changed the positions for the 'perpage' and the 'pagenumber' from my initial post. ie.

It used to be "f123-2-15.html" it is now: "f123-15-2.html"

where 2 is the page number, 15 is 'how many' per page. I changed both the rewriterules as well as the templates to allow me adding the navigation link hack. Please refer to my original post.

[lierduh]

Originally posted by cyc


I've uninstalled the changes that relate to this post, people started complaining about dead links


thanks

Please see my recent post regarding 'perpage' and 'pagenumber'.

For all this trouble, Google rewards by sending a bunch of bots everyday hammering the site.

BTW, I selected this hack instead of the archive hack because when visitors click on the Google results, they are taken straight to the forums instead of the archive pages.

[Erwin]

lierduh, I've had your system running on my site for the past few months now... works great - Google has added thousands of my pages into its archive.

[cyc]

Originally posted by lierduh


Please see my recent post regarding 'perpage' and 'pagenumber'.

For all this trouble, Google rewards by sending a bunch of bots everyday hammering the site.

BTW, I selected this hack instead of the archive hack because when visitors click on the Google results, they are taken straight to the forums instead of the archive pages.

I've tried making your suggested changes 3 or 4 times this morning, it works nice except the page links on a thread that spans more than one page.

The same page nav works fine when on the page with the thread headings, just not the threads themselves. BTW i've uninstalled it again cause me users hate dead links :laugh: I'm not using .htaccess because i run the server its on and therefore can edit the conf file.

cheers!

[lierduh]

Originally posted by cyc


I've tried making your suggested changes 3 or 4 times this morning, it works nice except the page links on a thread that spans more than one page.

The same page nav works fine when on the page with the thread headings, just not the threads themselves. BTW i've uninstalled it again cause me users hate dead links :laugh: I'm not using .htaccess because i run the server its on and therefore can edit the conf file.

cheers!

You were close.

There is no difference for the Rewriterules between httpd.conf and .htaccess

There are only two areas that could go wrong, that is the templates which generate the URL and the Rewriterule that catches/matches your URL. Again, if you could post the Dead URL and your rewriterules, I might be able to pin point what you did wrong.

[cyc]

Originally posted by lierduh


You were close.

There is no difference for the Rewriterules between httpd.conf and .htaccess

There are only two areas that could go wrong, that is the templates which generate the URL and the Rewriterules that catches/matches your URL. Again, if you could post the Dead URL and your rewriterules, I might be able to pin point what you did wrong.

I'll reapply the hack now

##### edit ######

I've got it working, my rewrite rules were alittle messed up.

Thanks for your help, I really appreciate it mate!

*waits for google*

cheers!

[lierduh]

Originally posted by Erwin
lierduh, I've had your system running on my site for the past few months now... works great - Google has added thousands of my pages into its archive.

Erwin, the original hack is created by fastforward, not by me. I merely posted my own experience (see last page) and I believe I might have made some improvement over the origian hack.

[lierduh]

Originally posted by cyc


I've got it working, my rewrite rules were alittle messed up.

Thanks for your help, I really appreciate it mate!

*waits for google*

cheers!

No worries, mate. (some downunder talking there)
Glad it is working for you.

Don't forget the last hack (RewriteCond) regarding the memberlist. Otherwise your member list navigation will not work. Fortunately this is a simple one, only need to add two lines to your rewriterules.

Waiting for more things to break, I am sure there is. eg. Merge Thread does not work anymore due to the new URL. Do I care to fix this is another question.

[Erwin]

When I say "your" system, lierduh, I mean the multiple page version of this hack. I don't use the same hack as you, I've customised my own version.