Forum Site Map for Search Engines [Archive]

View Full Version : Forum Site Map for Search Engines

Webdork

11-03-2002, 10:00 PM

Just a very quick hack that will print a static php page listing your categories and topics.

Search engines like Google wont index a dynamic URL off a dynamic URL so by providing a static page listing your topics dynamic URL's you will get a deeper crawl. I know there are different approaches on this board to optimizing for Google, but try this as well. It works :)

I then just create a link from the home page to the this file (you should try and name it sitemap.php or something similar.

Please note that I have deliberately not used any vbulletin templates etc... the goal of this page is to be as simple and clean as possible for crawlers like Google and AlltheWeb to index as much as possible.

Example of output here: http://www.sizematters.com.au/forums/sitemap.php

Just cut and paste the code and play. In another version I made it so the first 100 characters of the first post in each topic was created, but I dont think it helps much.

<?php require('./global.php');

// This file can be saved as any name, but upload to your forum dir.

// Only real variables for you to change. Include the trailing /

$siteurl = "http://www.sizematters.com.au/forums";
$sitename = "SizeMatters";

// You can safely edit the header and footer as well.

?>

<html>

<head>
<title>SizeMatters Forum Site Map</title>
<meta name="description" content="">
<meta name="keywords" content="">

<style>

</style>

</head>

<body>

<?php

echo "\r\n<h1>$sitename Forum Site Map</h1>\r\n";

echo "\r\n<p>Click here to return to <a href='$siteurl/'>$sitename</a></p>\r\n";

echo "\r\n<h2>$sitename Forum Categories</h2>\r\r\n";

$result = mysql_query("SELECT title, forumid, description FROM forum ORDER BY forumid");

echo "<b>Number of Forums: </b>".mysql_num_rows($result)."<br><br>\r\n";

while($row = mysql_fetch_assoc($result))
{
echo "<a href='$siteurl/forumdisplay.php?s=&forumid=".$row["forumid"]."'>".$row["title"]."</a> - ".$row["description"]."<br>\r\n";

}

echo "\r\n<h2>$sitename Most Recent Forum Topics</h2>\r\r\n";

$result = mysql_query("SELECT title, threadid FROM thread ORDER BY threadid LIMIT 20");

echo "<b>20 Most Recent Topics</b><br><br>\r\n";

while($row = mysql_fetch_assoc($result))
{
echo "<a href='$siteurl/showthread.php?s=&threadid=".$row["threadid"]."'>".$row["title"]."</a><br>\r\n";

}

echo "\r\n<h2>$sitename Forum Topics</h2>\r\r\n";

$result = mysql_query("SELECT title, threadid FROM thread ORDER BY title");

echo "<b>Number of Topics: </b>".mysql_num_rows($result)."<br><br>\r\n";

while($row = mysql_fetch_assoc($result))
{
echo "<a href='$siteurl/showthread.php?s=&threadid=".$row["threadid"]."'>".$row["title"]."</a><br>\r\n";
}

?>

</body>

</html>

If you do make some changes that you think work well - please share :)

NOTE: This version WILL show contents of private forums if you have them. If you have certain forums you dont want to have their content listed - see the revised code a couple of posts below.

Velocd

11-04-2002, 03:56 AM

Nice idea, ofcourse for a larger forum the listing of threads can get very long ;)

One thing though, the use of mysql_query could have been replaced with $DB_site->query, since it's specified for vBulletin.

Velocd

11-04-2002, 04:03 AM

Excellent hack, and you did a nifty job in the CSS of it as well.
Here is another demo for anybody wondering:
http://www.diffusion4.com/sitemap.php

Webdork

11-04-2002, 04:11 AM

Spankyou :)

If the topics get too long it wouldnt be too hard to put in a previous / next link - but that would again run into the problem of dunamic url off a dynamic url :)

Smoothie

11-04-2002, 04:26 AM

this shows the topics of private forums, is it suppose to?

Webdork

11-04-2002, 04:28 AM

Shows everything. I hadnt thought about pvt forums. Could add a bit of code where it wouldnt.

Webdork

11-04-2002, 04:36 AM

I dont have any pvt forums so not sure how they are flagged so cant see anything obvious to edit for now.

Velocd

11-04-2002, 04:50 AM

You have a small in the listing of the top 20 recent-threads, as you forgot to order it by dateline.

#1. Find:

$result = mysql_query("SELECT title, threadid FROM thread ORDER BY threadid LIMIT 20");

Replace:
$result = mysql_query("SELECT title, threadid, dateline FROM thread ORDER BY dateline DESC LIMIT 20");

;)

Webdork

11-04-2002, 05:01 AM

Thanks - I initially had it for some reason but took it out...

If you want to EXCLUDE some categories - eg private areas use this code:

<?php require('./global.php');

// This file can be saved as any name, but upload to your forum dir.

// Only real variables for you to change. Include the trailing /

$siteurl = "http://www.sizematters.com.au/forums";
$sitename = "SizeMatters";

/* Slightly more complicated - if you have forums you dont want shown find this line (it appears 3 times)
WHERE forumid NOT IN ('18')
and replace the 18 with the forum you DONT want displayed. In my case its my forum 18. If you have multiple
topics you dont want shown then the format is like:
WHERE forumid NOT IN ('18', '19','20')
etc...
If you dont have any that you want to hide then delete the code
WHERE forumid NOT IN ('18')

(If anyone wants to make this a variable or IF statement it would be appreciated :)

*/

// You can safely edit the header and footer as well.

?>

<html>

<head>
<title>SizeMatters Forum Site Map</title>
<meta name="description" content="">
<meta name="keywords" content="">

<style>

</style>

</head>

<body>

<?php

echo "\r\n<h1>$sitename Forum Site Map</h1>\r\n";

echo "\r\n<p>Click here to return to <a href='$siteurl/'>$sitename</a></p>\r\n";

echo "\r\n<h2>$sitename Forum Categories</h2>\r\r\n";

$result = mysql_query("SELECT forumid, title, description FROM forum WHERE forumid NOT IN ('18') ORDER BY forumid");

echo "<b>Number of Forums: </b>".mysql_num_rows($result)."<br><br>\r\n";

while($row = mysql_fetch_assoc($result))
{
echo "<a href='$siteurl/forumdisplay.php?s=&forumid=".$row["forumid"]."'>".$row["title"]."</a> - ".$row["description"]."<br>\r\n";

}

echo "\r\n<h2>$sitename Most Recent Forum Topics</h2>\r\r\n";

$result = mysql_query("SELECT title, threadid, dateline FROM thread WHERE forumid NOT IN ('18')ORDER BY dateline DESC LIMIT 20");

echo "<b>20 Most Recent Topics</b><br><br>\r\n";

while($row = mysql_fetch_assoc($result))
{
echo "<a href='$siteurl/showthread.php?s=&threadid=".$row["threadid"]."'>".$row["title"]."</a><br>\r\n";

}

echo "\r\n<h2>$sitename Forum Topics</h2>\r\r\n";

$result = mysql_query("SELECT threadid, title, threadid FROM thread WHERE forumid NOT IN ('18') ORDER BY title");

echo "<b>Number of Topics: </b>".mysql_num_rows($result)."<br><br>\r\n";

while($row = mysql_fetch_assoc($result))
{
echo "<a href='$siteurl/showthread.php?s=&threadid=".$row["threadid"]."'>".$row["title"]."</a><br>\r\n";
}

?>

</body>

</html>

Webdork

11-04-2002, 05:28 AM

Because it uses the same code layout - heres a REALLY quick and dodgy hack to display Private Messages

<?PHP require('./global.php');?>

<html>
<head>
<title>Display pvt messages</title>
<meta name="description" content="">
<meta name="keywords" content="">

<style>

</style>

</head>

<body>

<?PHP

/* REALLY dodgy script that shows pvt messages.
I haven't spent any time refining this so play with it if you will.
Again simply save to your forums directory.
You might like to change it so it adds author id's but I was simply more curious
as to what was being discussed not necessarily by who.
Brought to you by http://www.phphacks.com ;)
*/

echo "<h1>Private Messages</h1>";

$result = mysql_query("SELECT DISTINCT message, title FROM privatemessage ORDER BY title");

echo "<b>Number of Pvt Messages: </b>".mysql_num_rows($result)."<br><br>\r\n";

while($row = mysql_fetch_assoc($result))

{
echo "<b>".$row["title"]."</b><br>".$row["message"]."<br><br>\r\n";
}

?>

</body>
</html>

Erwin

11-04-2002, 08:56 AM

I suggest you place the php code in a text file, so that it is hidden from unlicensed members, and so that members can download it onto their hard disk. Good idea, btw. :)

MaXxed

11-04-2002, 02:53 PM

Works well. Thanks!

Max.

djr

11-04-2002, 06:01 PM

Is there a way with META NAME keywords to force Google not to index the forums, but only to index this file?

Something in the likes of:
META NAME="ROBOTS" CONTENT="INDEX,NOFOLLOW"

and in the normal forums:
META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW"

or wouldn't that make a difference?

If so, where would I place the META NAME? In the header template?

Webdork

11-04-2002, 07:10 PM

Well - you would simply use a robots.txt file - much easier in my opinion, but I would never restrict Google from trying to index anything.

webhost

11-09-2002, 10:40 PM

What if you have some forums that you still have in your control panel but you have them turned off to not show on your forum. Code still shows them. How to exclude those types of forums from the map?

99SIVTEC

11-10-2002, 06:02 PM

Works great! I have it installed on both of my sites.

CRego3D

11-18-2002, 04:49 AM

is there a way to make this work off a template ? so we can include $header and $footer ? :)

Automated

11-23-2002, 07:05 PM

How would i go about making the page adobt the header and footer templates so it looks like a normal page not just a doorway page?

Thanks
Matthew

Webdork

11-28-2002, 11:30 PM

I made it deliberately so that you wouldnt have headers and footers - or anything to distract the search engines from their job of indexing :)

Having said that though a header and footer can EASILY be added. Its just a html file after all. Add your header and footer before and after the php code.

corn

12-08-2002, 08:45 PM

Looks great man !!

MaXxed

12-08-2002, 10:37 PM

Google has posted some new guidelines, and one point about site maps is that they should contain less than 100 links. I wonder if this can be modified to spread over a number of pages?

nintendo

02-08-2003, 09:39 PM

With so many links on one pages, this makes it look like a Linkfarm, and search engines don't like those. Listing 50 posts at a time would make the Googlebot more happy.

Webdork

02-10-2003, 10:42 PM

I havent had any problems with over 1,000 links on a page :)

Depends on your PR as to how many will get crawled...

99SIVTEC

02-17-2003, 10:33 PM

well it definetly worked on my sites. I have tons of hits from the sitemap everyday, but I checked my server log and JUST the sitemap.php file is using almost 3GB of bandwidth a month. Any way to gzip the output and possibly shorten it to only include a hundred or so links?

Brain Crusher

02-17-2003, 11:10 PM

can i make it also for only categories

or only the newest 50 topics?

99SIVTEC

03-04-2003, 03:41 AM

I checked and even your sitemap (the one on the 1st page) is almost 1MB in size. That's a ton of bandwidth.

gmarik

05-24-2003, 11:22 AM

I dropped the file in vbfolder/sitemap/index.php and changed ../global.php but nothing seemed to work. Any idea why?