The Arcive of Official vBulletin Modifications Site.

About Developer · **Released**: 04-29-2001

I am tired of my 200,000 posts not being listed in Google. I was inspired by phpbuilder.com this morning and I wrote:

vbSpiderFriend - the search engine indexer for all of your posts

Purpose: Allow search engine spiders to crawl a linked list of all of your posts.

Project Requirements:

-Friendly URLs (no query strings)
-Good dynamic meta tags
-Never have to touch the script again.. It is Y3K compliant, simply re-submit to the engines to update your listings

Install Requirements:

-vBulletin 1.x or 2.x
-about 10 minutes

1) Download the attached Zip.

2) Open class.mysql.php and put your database login info at the top.

3) Create a new directory called archive under your forum, like /forum/archive

4) Open the included .htaccess and change the Error 404 to your new archive path.

5) Open index.php and change the self-explanatory variables at the top of the file.

6) Upload all 3 files to your archive directory.

7) Submit /forum/archive/index.php to search engines and watch em crawl

DISCLAIMER: I don't use 2.x but I checked the schema and this should work fine.

NOTES: This uses ErrorDocument and query string parsing to get the variables needed. I do not have the time or energy to troubleshoot this if it does not work on your server. Sorry!

Gilby · #52 05-14-2001, 03:55 PM

Quote:

Originally posted by Overgrow
This is never meant to be viewed by a user except when they click through the search engine to the plain-text thread.

How about doing a redirect for users? So when they click on the link from a search engine like google, the script will redirect them to the vbulletin page that has that thread instead of showing them the page that isn't formatted as much.

Overgrow · #53 05-14-2001, 04:02 PM

Oooh yea ya know that thought had crossed my mind.. then I quickly smoked it out of there.

I wondered how spiders treat redirections? Will it piss them off if it's always redirected and get the site delisted? Do they follow javascript hrefs? Should I use meta or jscript or both?

Help me out on how the spiders will handle this and I will get it implemented quickly.. I think it's a great idea.

Good point also is that it won't matter when you fix that-- the search engines will have your info and if you put a redirect in, all further links in will use it.

#54 05-14-2001, 04:05 PM

Most spiders ignore pages with redirects.

Gilby · #55 05-14-2001, 04:10 PM

Quote:

Originally posted by Overgrow
I wondered how spiders treat redirections? Will it piss them off if it's always redirected and get the site delisted? Do they follow javascript hrefs? Should I use meta or jscript or both?

I was thinking of a redirect from the script side, it will check out the referrer, and if it is from a link external to the site (such as from a search engine), then it will execute the php code to redirect it, so it'd be:

PHP Code:


			
header("Location:/forums/showthread.php?threadid=1234");

Overgrow · #56 05-14-2001, 05:10 PM

Ok... it's easy to detect if the Referer is from your host. Getting the header(location

code to work is being a bugger! I've used this many times before.. I know the error that happens if you sent previous headers, but using the header(location

here just fails! So I used both metas and jscript refresh.

Try this index.php and let me know if it works for you before I make it an overall change in the main zip. You won't notice any difference if you follow a link down the tree like a spider. IF you copy and paste out that final thread URL, go to a new site, then paste that URL back in then you should be forwarded to the real thread...

ie, if you come through a search engine, and not from your own site, the user should be pushed to the real deal

Gilby · #57 05-14-2001, 05:27 PM

Quote:

Originally posted by Overgrow
ie, if you come through a search engine, and not from your own site, the user should be pushed to the real deal

Didn't work. I got:
Warning: Cannot add header information - headers already sent by (output started at
/home/sites/site3/web/forums/archive/index.php:88) in /home/sites/site3/web/forums/archive/index.php on line 316

Also, when I got that error, it redirected me to the real thread, which it's not supposed to. Make sure that when the referrer is blank, that it doesn't redirect. It should only redirect when the referrer is defined and is from an external link.

Overgrow · #58 05-14-2001, 05:47 PM

This is the if statement

PHP Code:


			
if ((!strstr(getenv(HTTP_REFERER),$homeURL)) or (strlen(getenv(HTTP_REFERER)) < 1)) {

It is checking that the referer is greater than 0 length. This does work for me

Can you troubleshoot it a bit on your end?

Gilby · #59 05-14-2001, 05:57 PM

Quote:

Originally posted by Overgrow
This is the if statement

PHP Code:

if ((!strstr(getenv(HTTP_REFERER),$homeURL)) or (strlen(getenv(HTTP_REFERER)) < 1)) {

It is checking that the referer is greater than 0 length. This does work for me Can you troubleshoot it a bit on your end?

Looking at that line, I see the problem. strstr() is case sensitive, and I defined my $homeURL to be a different case than what I accessed it from, so you should change that to use stristr() instead.

Overgrow · #60 05-14-2001, 06:16 PM

Thanks! I changed to stristr, good point. As soon as you let me know that this works I will make the changes in the main download-zip. That is a new index.zip up there a few posts back (with stristr).

steven · #61 05-15-2001, 01:04 AM

This is one awesome hack, the only problem that I am having is that the spider is accessing forums that I have specified it not to. Is there a quick fix for this?

Below is my index.php file

Code:

<?

/*

vbSpiderFriend v0.1a by ~shabang~

** Free License: YOU MUST LEAVE THE FOOTER INTACT **

*/

$privateForums="|4|5||6||7||8||9||33|"; // Hidden forumids, enclosed by | pipes
$firstPost="07/04/2000"; // MM/DD/YYYY of your forum's first post
$spacer="&nbsp;&nbsp;"; // The characters or spaces to use as one indent
$forumURL="/v2"; // Base URL of your forum

$homeURL="http://www.baddealings.com"; // The link URL for the top of the page
$homeLink="Baddealings.com - Online Source For Consumer Complaints"; // The link text for the top of the page

$keywords="complaints, complain, consumer complaints, consumer, fraud, scams, consumer fraud, grievances, auctions, electronics, feedback, shopping, buyers/sellers, forums, community, ripped off, chat";  // SET META INFORMATION HERE
$description="Baddealings.com provides an arena for members to complain about a product or service. Search our online database for the lastest information on consumer complaints and scams.";			 // The script will add to these fields
$pagetitle="Baddealings.com - Online Source For Consumer Complaints";					 // with the info from the thread

include("class.mysql.php");



// NOTHING MORE TO EDIT BELOW /////////////////////////////////////////////////////////////




$baseURL=$forumURL."/archive";
$dateSplit = split("/",$firstPost);
$firstMonth = preg_replace("/^0/","",$dateSplit[0]);
$firstYear = $dateSplit[2];
$currentYear=date("Y",time());
$currentMonth=preg_replace("/^0/","",date("m",time()));


// setup the DB connection

$db = new MySQL;

if (!$db->init()) {
	echo "no DB connection<br>";
	exit;
}



// parse query string

if (ereg("/archive/[0-9]",getenv('REQUEST_URI'))) {

	$urlArray=explode("/",getenv('REQUEST_URI'));  //split the URL path 
	$a=0;

	while($urlArray[$a] != "archive") {
		$a++;
	}

	$forumID=$urlArray[($a+1)];
	$year=$urlArray[($a+2)];
	$month=$urlArray[($a+3)];
	$week=$urlArray[($a+4)];
	$threadID=$urlArray[($a+5)];

	if (eregi("[a-z]",$forumID) or $forumID < 1 or strstr($privateForums,"|".$forumID."|"))
		$forumID=1;
}

header("Status: 200 OK"); 

//echo "<br>&nbsp;<br>$forumID - $year - $month - $week - $threadID";

if (empty($forumID)) {
	htmlHeader();
	forumList();
	exit;
} else if (empty($year) or empty($month)) {
	htmlHeader();
	weekList();
} else if (empty($threadID)) {
	htmlHeader();
	threadList();
} else {
	showThread();
}

echo "<center><font size=1><br>&nbsp;<br>vbSpiderFriend by ~shabang~ of <a href=\"http://www.overgrow.com/\">Overgrow</a></font></center></body></html>";


function showThread() {
	
	global $db,$baseURL,$forumID,$threadID,$homeURL,$homeLink,$forumURL,$keywords,$description,$pagetitle;

	$query = "SELECT thread.title as ttitle,forum.title as ftitle FROM thread LEFT JOIN forum ON thread.forumid=forum.forumid WHERE thread.threadid='$threadID'";

	$data = $db->select($query);

	if(!empty($data)) {

		while ( list ( $key,$forum ) = each ($data)) {

			$forumline = "Forum: <a href=\"$forumURL/forumdisplay.php?forumid=$forumID\">$forum[ftitle]</a><br>";
			$threadline = "Thread: <a href=\"$forumURL/showthread.php?threadid=$threadID\"><b>$forum[ttitle]</a></b><br>";
			$threadtitle = $forum[ttitle];
		}
	} else {
		echo $query;
	}

	$ks = str_replace(" ",",",$threadtitle);

	$keywords.=",$ks";
	$description.=": $threadtitle";
	$pagetitle.=" : $threadtitle";
	htmlHeader();

	echo "<center><a href=\"$homeURL\"><b>$homeLink</b></a><br>&nbsp;</center><br>";

	echo spacer(1).$forumline;
	echo spacer(2).$threadline;
	
	$data=$db->select("SELECT post.dateline as dateline,post.postid as postid,post.pagetext as pagetext,post.username as fakename,
								post.title as title,post.userid as userid, user.userid as userid,user.username as username,
								user.usertitle as usertitle,user.posts as posts									
								FROM post 
								LEFT JOIN user ON (user.userid = post.userid)
								WHERE post.threadid=$threadID AND visible=1							
								ORDER BY dateline");

	if(!empty($data)) {
		
		echo "<table border=0 cellpadding=10>";

		while ( list ( $key,$posts ) = each ($data)) {

			$username=$posts[username];

			if(empty($username))
				$username=$posts[fakename];

			echo "<tr><td valign=top>$username</td><td>".nl2br($posts[pagetext])."</td></tr>";
		}

		echo "</table>";
	}

}



function threadList(){

	global $db,$baseURL,$forumID,$year,$month,$week;

	echo "<a href=\"$baseURL/index.php\">Archives</a><br>";

	$query = "SELECT title FROM forum WHERE forumid='$forumID'";

	$data = $db->select($query);

	if(!empty($data)) {

		while ( list ( $key,$forum ) = each ($data)) {

			echo spacer(1)."<a href=\"$baseURL/$forumID\">$forum[title]</a><br>";
		}
	}

	switch($week) {
	
		case("1"):
			$fw = "01";
			$lw = "07";
		break;
		case("2"):
			$fw = "08";
			$lw = "14";
		break;
		case("3"):
			$fw = "15";
			$lw = "21";
		break;
		case("4"):
			$fw = "22";
			$lw = "31";
		break;

		default:
			$fw = "01";
			$lw = "07";
		
	}

	$date1 = "$month/$fw/$year";
	$date2 = "$month/$lw/$year";

	$ts1 = strtotime("$date1");
	$ts2 = strtotime("$date2");

	echo spacer(2)."Dates: $date1 to $date2<br>&nbsp;<br>";

	$query = "SELECT title,threadid,lastpost FROM thread WHERE lastpost > '$ts1' AND lastpost < '$ts2' AND forumid='$forumID' ORDER BY dateline ASC";
	
	$data = $db->select($query);

	if(!empty($data)) {

		while ( list ( $key,$thread ) = each ($data)) {

			echo spacer(3)."<a href=\"$baseURL/$forumID/$year/$month/$week/$thread[threadid]\">$thread[title]</a><br>";
			//echo spacer(3)."<a href=\"http://www.overgrow.com/edge/showthread.php?threadid=$thread[threadid]\">$thread[title]</a><br>";
		}
	} else {
		echo spacer(3)."No posts, please <a href=\"".getenv(HTTP_REFERER)."\">go back</a>.";
	}

}

function spacer($n) {

	global $spacer;

	while($n!=0){
		$return.=$spacer;
		$n--;
	}
	return $return;
}

function removeZero($n) {

	preg_replace ("/./","",$n);
	return $n;
}


function weekList() {

	global $db,$baseURL,$forumID,$firstYear,$currentYear,$currentMonth,$firstMonth;

	$months = array("blank","January","February","March","April","May","June","July","August","September","October","November","December");

	echo "<a href=\"$baseURL/index.php\">Archives</a><br>";

	$query = "SELECT title FROM forum WHERE forumid='$forumID'";

	$data = $db->select($query);

	if(!empty($data)) {

		while ( list ( $key,$forum ) = each ($data)) {

			echo spacer(1)."$forum[title]<br>";
		}
	}

	for($y=$firstYear;$y<=$currentYear;$y++) {

		echo spacer(2)."$y<br>&nbsp;<br>";

		if($y==$firstYear)
			$fm=$firstMonth;
		else
			$fm=1;

		if($y==$currentYear)
			$lm=($currentMonth + 1);
		else
			$lm=13;
			

		for($m=$fm;$m<$lm;$m++){
			
			echo spacer(3).$months[$m]."<br>";

			for($w=1;$w<5;$w++) {

				if (strlen($m) < 2)
					$mo="0".$m;
				else
					$mo=$m;

				echo spacer(4)."<a href=\"$baseURL/$forumID/$y/$mo/$w\">week $w</a><br>";
			}

			echo "&nbsp;<br>";
		}
	}

}


function forumList() {

	global $db,$baseURL,$privateForums;

	echo "Archives<br>";

	$privateForums=preg_replace("/^\|/","",$privateForums);
	$privateForums=preg_replace("/\|$/","",$privateForums);

	$pfs = explode("|",$privateForums);
	$whereclause="";
	$wherecounter=0;


	while($pf=array_shift($pfs)) {
		if(ereg("[0-9]",$pf)) {

			if($wherecounter==0) {
				$whereclause= " WHERE forumid!='$pf'";
				$wherecounter++;
			} else {
				$whereclause .= " AND forumid!='$pf'";
			}
		}
	 }
	
	

	$query = "SELECT title,forumid FROM forum$whereclause ORDER BY forumid ASC";

	$data = $db->select($query);

	if(!empty($data)) {

		while ( list ( $key,$forum ) = each ($data)) {

			echo spacer(1)."<a href=\"$baseURL/$forum[forumid]\">$forum[title]</a><br>";
		}
	}	
}


function htmlHeader() {

	global $keywords,$description,$pagetitle;

?>

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML>
<HEAD>
<TITLE> <?=$pagetitle?> </TITLE>
<META NAME="Generator" CONTENT="EditPlus">
<META NAME="Author" CONTENT="">
<META NAME="Keywords" CONTENT="<?=$keywords?>">
<META NAME="Description" CONTENT="<?=$description?>">

<style type="text/css">

BODY {font: 10pt verdana,arial,sans-serif;}
TD {font: 10pt verdana,arial,sans-serif;}

</style>

</HEAD>

<BODY bgcolor="#ffffff">

<?
}

?>

Thanks Steven

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.04600 seconds Memory Usage 2,338KB Queries Executed 25 (?)
More Information
Template Usage: (1)SHOWTHREAD (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (1)ad_showthread_beforeqr (1)bbcode_code (3)bbcode_php (4)bbcode_quote (1)footer (1)forumjump (1)forumrules (1)gobutton (1)header (1)headinclude (1)modsystem_post (1)navbar (6)navbar_link (120)option (1)pagenav (1)pagenav_curpage (4)pagenav_pagelink (1)pagenav_pagelinkrel (11)post_thanks_box (11)post_thanks_button (1)post_thanks_javascript (1)post_thanks_navbar_search (11)post_thanks_postbit_info (10)postbit (10)postbit_onlinestatus (11)postbit_wrapper (1)spacer_close (1)spacer_open (1)tagbit_wrapper Phrase Groups Available: global inlinemod postbit posting reputationlevel showthread	Included Files: ./showthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/functions_bigthree.php ./includes/class_postbit.php ./includes/class_bbcode.php ./includes/functions_reputation.php ./includes/functions_post_thanks.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete showthread_start showthread_getinfo forumjump showthread_post_start showthread_query_postids showthread_query bbcode_fetch_tags bbcode_create showthread_postbit_create postbit_factory postbit_display_start post_thanks_function_post_thanks_off_start post_thanks_function_post_thanks_off_end post_thanks_function_fetch_thanks_start post_thanks_function_fetch_thanks_end post_thanks_function_thanked_already_start post_thanks_function_thanked_already_end fetch_musername postbit_imicons bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete postbit_display_complete post_thanks_function_can_thank_this_post_start pagenav_page pagenav_complete tag_fetchbit_complete forumrules navbits navbits_complete showthread_complete
Messages:

The Arcive of Official vBulletin Modifications Site.

It is not a VB3 engine, just a parsed copy!