Go Back   vb.org Archive > vBulletin Modifications > Archive > vB.org Archives > vBulletin 2.x > vBulletin 2.x Full Releases
FAQ Community Calendar Today's Posts Search

Reply
 
Thread Tools
Details »»

Version: , by Overgrow Overgrow is offline
Developer Last Online: Jun 2004 Show Printable Version Email this Page

Version: 2.0.x Rating:
Released: 04-29-2001 Last Update: Never Installs: 53
 
No support by the author.

I am tired of my 200,000 posts not being listed in Google. I was inspired by phpbuilder.com this morning and I wrote:

vbSpiderFriend - the search engine indexer for all of your posts

Purpose: Allow search engine spiders to crawl a linked list of all of your posts.

Project Requirements:

-Friendly URLs (no query strings)
-Good dynamic meta tags
-Never have to touch the script again.. It is Y3K compliant, simply re-submit to the engines to update your listings

Install Requirements:

-vBulletin 1.x or 2.x
-about 10 minutes


1) Download the attached Zip.

2) Open class.mysql.php and put your database login info at the top.

3) Create a new directory called archive under your forum, like /forum/archive

4) Open the included .htaccess and change the Error 404 to your new archive path.

5) Open index.php and change the self-explanatory variables at the top of the file.

6) Upload all 3 files to your archive directory.

7) Submit /forum/archive/index.php to search engines and watch em crawl


DISCLAIMER: I don't use 2.x but I checked the schema and this should work fine.

NOTES: This uses ErrorDocument and query string parsing to get the variables needed. I do not have the time or energy to troubleshoot this if it does not work on your server. Sorry!

Show Your Support

  • This modification may not be copied, reproduced or published elsewhere without author's permission.

Comments
  #52  
Old 05-14-2001, 03:55 PM
Gilby Gilby is offline
 
Join Date: Oct 2001
Posts: 173
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally posted by Overgrow
This is never meant to be viewed by a user except when they click through the search engine to the plain-text thread.
How about doing a redirect for users? So when they click on the link from a search engine like google, the script will redirect them to the vbulletin page that has that thread instead of showing them the page that isn't formatted as much.
Reply With Quote
  #53  
Old 05-14-2001, 04:02 PM
Overgrow's Avatar
Overgrow Overgrow is offline
 
Join Date: Nov 2001
Posts: 320
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Oooh yea ya know that thought had crossed my mind.. then I quickly smoked it out of there.

I wondered how spiders treat redirections? Will it piss them off if it's always redirected and get the site delisted? Do they follow javascript hrefs? Should I use meta or jscript or both?

Help me out on how the spiders will handle this and I will get it implemented quickly.. I think it's a great idea.

Good point also is that it won't matter when you fix that-- the search engines will have your info and if you put a redirect in, all further links in will use it.
Reply With Quote
  #54  
Old 05-14-2001, 04:05 PM
eiko
Guest
 
Posts: n/a
Default

Most spiders ignore pages with redirects.
Reply With Quote
  #55  
Old 05-14-2001, 04:10 PM
Gilby Gilby is offline
 
Join Date: Oct 2001
Posts: 173
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally posted by Overgrow
I wondered how spiders treat redirections? Will it piss them off if it's always redirected and get the site delisted? Do they follow javascript hrefs? Should I use meta or jscript or both?
I was thinking of a redirect from the script side, it will check out the referrer, and if it is from a link external to the site (such as from a search engine), then it will execute the php code to redirect it, so it'd be:
PHP Code:
header("Location:/forums/showthread.php?threadid=1234"); 
Reply With Quote
  #56  
Old 05-14-2001, 05:10 PM
Overgrow's Avatar
Overgrow Overgrow is offline
 
Join Date: Nov 2001
Posts: 320
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Ok... it's easy to detect if the Referer is from your host. Getting the header(location code to work is being a bugger! I've used this many times before.. I know the error that happens if you sent previous headers, but using the header(location here just fails! So I used both metas and jscript refresh.

Try this index.php and let me know if it works for you before I make it an overall change in the main zip. You won't notice any difference if you follow a link down the tree like a spider. IF you copy and paste out that final thread URL, go to a new site, then paste that URL back in then you should be forwarded to the real thread...

ie, if you come through a search engine, and not from your own site, the user should be pushed to the real deal
Reply With Quote
  #57  
Old 05-14-2001, 05:27 PM
Gilby Gilby is offline
 
Join Date: Oct 2001
Posts: 173
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally posted by Overgrow
ie, if you come through a search engine, and not from your own site, the user should be pushed to the real deal
Didn't work. I got:
Warning: Cannot add header information - headers already sent by (output started at
/home/sites/site3/web/forums/archive/index.php:88) in /home/sites/site3/web/forums/archive/index.php on line 316

Also, when I got that error, it redirected me to the real thread, which it's not supposed to. Make sure that when the referrer is blank, that it doesn't redirect. It should only redirect when the referrer is defined and is from an external link.
Reply With Quote
  #58  
Old 05-14-2001, 05:47 PM
Overgrow's Avatar
Overgrow Overgrow is offline
 
Join Date: Nov 2001
Posts: 320
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

This is the if statement

PHP Code:
if ((!strstr(getenv(HTTP_REFERER),$homeURL)) or (strlen(getenv(HTTP_REFERER)) < 1)) { 
It is checking that the referer is greater than 0 length. This does work for me Can you troubleshoot it a bit on your end?
Reply With Quote
  #59  
Old 05-14-2001, 05:57 PM
Gilby Gilby is offline
 
Join Date: Oct 2001
Posts: 173
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally posted by Overgrow
This is the if statement

PHP Code:
if ((!strstr(getenv(HTTP_REFERER),$homeURL)) or (strlen(getenv(HTTP_REFERER)) < 1)) { 
It is checking that the referer is greater than 0 length. This does work for me Can you troubleshoot it a bit on your end?
Looking at that line, I see the problem. strstr() is case sensitive, and I defined my $homeURL to be a different case than what I accessed it from, so you should change that to use stristr() instead.
Reply With Quote
  #60  
Old 05-14-2001, 06:16 PM
Overgrow's Avatar
Overgrow Overgrow is offline
 
Join Date: Nov 2001
Posts: 320
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Thanks! I changed to stristr, good point. As soon as you let me know that this works I will make the changes in the main download-zip. That is a new index.zip up there a few posts back (with stristr).
Reply With Quote
  #61  
Old 05-15-2001, 01:04 AM
steven steven is offline
 
Join Date: Oct 2001
Posts: 10
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

This is one awesome hack, the only problem that I am having is that the spider is accessing forums that I have specified it not to. Is there a quick fix for this?

Below is my index.php file

Code:
<?

/*

vbSpiderFriend v0.1a by ~shabang~

** Free License: YOU MUST LEAVE THE FOOTER INTACT **

*/

$privateForums="|4|5||6||7||8||9||33|"; // Hidden forumids, enclosed by | pipes
$firstPost="07/04/2000"; // MM/DD/YYYY of your forum's first post
$spacer="&nbsp;&nbsp;"; // The characters or spaces to use as one indent
$forumURL="/v2"; // Base URL of your forum

$homeURL="http://www.baddealings.com"; // The link URL for the top of the page
$homeLink="Baddealings.com - Online Source For Consumer Complaints"; // The link text for the top of the page

$keywords="complaints, complain, consumer complaints, consumer, fraud, scams, consumer fraud, grievances, auctions, electronics, feedback, shopping, buyers/sellers, forums, community, ripped off, chat";  // SET META INFORMATION HERE
$description="Baddealings.com provides an arena for members to complain about a product or service. Search our online database for the lastest information on consumer complaints and scams.";			 // The script will add to these fields
$pagetitle="Baddealings.com - Online Source For Consumer Complaints";					 // with the info from the thread

include("class.mysql.php");



// NOTHING MORE TO EDIT BELOW /////////////////////////////////////////////////////////////




$baseURL=$forumURL."/archive";
$dateSplit = split("/",$firstPost);
$firstMonth = preg_replace("/^0/","",$dateSplit[0]);
$firstYear = $dateSplit[2];
$currentYear=date("Y",time());
$currentMonth=preg_replace("/^0/","",date("m",time()));


// setup the DB connection

$db = new MySQL;

if (!$db->init()) {
	echo "no DB connection<br>";
	exit;
}



// parse query string

if (ereg("/archive/[0-9]",getenv('REQUEST_URI'))) {

	$urlArray=explode("/",getenv('REQUEST_URI'));  //split the URL path 
	$a=0;

	while($urlArray[$a] != "archive") {
		$a++;
	}

	$forumID=$urlArray[($a+1)];
	$year=$urlArray[($a+2)];
	$month=$urlArray[($a+3)];
	$week=$urlArray[($a+4)];
	$threadID=$urlArray[($a+5)];

	if (eregi("[a-z]",$forumID) or $forumID < 1 or strstr($privateForums,"|".$forumID."|"))
		$forumID=1;
}

header("Status: 200 OK"); 

//echo "<br>&nbsp;<br>$forumID - $year - $month - $week - $threadID";

if (empty($forumID)) {
	htmlHeader();
	forumList();
	exit;
} else if (empty($year) or empty($month)) {
	htmlHeader();
	weekList();
} else if (empty($threadID)) {
	htmlHeader();
	threadList();
} else {
	showThread();
}

echo "<center><font size=1><br>&nbsp;<br>vbSpiderFriend by ~shabang~ of <a href=\"http://www.overgrow.com/\">Overgrow</a></font></center></body></html>";


function showThread() {
	
	global $db,$baseURL,$forumID,$threadID,$homeURL,$homeLink,$forumURL,$keywords,$description,$pagetitle;

	$query = "SELECT thread.title as ttitle,forum.title as ftitle FROM thread LEFT JOIN forum ON thread.forumid=forum.forumid WHERE thread.threadid='$threadID'";

	$data = $db->select($query);

	if(!empty($data)) {

		while ( list ( $key,$forum ) = each ($data)) {

			$forumline = "Forum: <a href=\"$forumURL/forumdisplay.php?forumid=$forumID\">$forum[ftitle]</a><br>";
			$threadline = "Thread: <a href=\"$forumURL/showthread.php?threadid=$threadID\"><b>$forum[ttitle]</a></b><br>";
			$threadtitle = $forum[ttitle];
		}
	} else {
		echo $query;
	}

	$ks = str_replace(" ",",",$threadtitle);

	$keywords.=",$ks";
	$description.=": $threadtitle";
	$pagetitle.=" : $threadtitle";
	htmlHeader();

	echo "<center><a href=\"$homeURL\"><b>$homeLink</b></a><br>&nbsp;</center><br>";

	echo spacer(1).$forumline;
	echo spacer(2).$threadline;
	
	$data=$db->select("SELECT post.dateline as dateline,post.postid as postid,post.pagetext as pagetext,post.username as fakename,
								post.title as title,post.userid as userid, user.userid as userid,user.username as username,
								user.usertitle as usertitle,user.posts as posts									
								FROM post 
								LEFT JOIN user ON (user.userid = post.userid)
								WHERE post.threadid=$threadID AND visible=1							
								ORDER BY dateline");

	if(!empty($data)) {
		
		echo "<table border=0 cellpadding=10>";

		while ( list ( $key,$posts ) = each ($data)) {

			$username=$posts[username];

			if(empty($username))
				$username=$posts[fakename];

			echo "<tr><td valign=top>$username</td><td>".nl2br($posts[pagetext])."</td></tr>";
		}

		echo "</table>";
	}

}



function threadList(){

	global $db,$baseURL,$forumID,$year,$month,$week;

	echo "<a href=\"$baseURL/index.php\">Archives</a><br>";

	$query = "SELECT title FROM forum WHERE forumid='$forumID'";

	$data = $db->select($query);

	if(!empty($data)) {

		while ( list ( $key,$forum ) = each ($data)) {

			echo spacer(1)."<a href=\"$baseURL/$forumID\">$forum[title]</a><br>";
		}
	}

	switch($week) {
	
		case("1"):
			$fw = "01";
			$lw = "07";
		break;
		case("2"):
			$fw = "08";
			$lw = "14";
		break;
		case("3"):
			$fw = "15";
			$lw = "21";
		break;
		case("4"):
			$fw = "22";
			$lw = "31";
		break;

		default:
			$fw = "01";
			$lw = "07";
		
	}

	$date1 = "$month/$fw/$year";
	$date2 = "$month/$lw/$year";

	$ts1 = strtotime("$date1");
	$ts2 = strtotime("$date2");

	echo spacer(2)."Dates: $date1 to $date2<br>&nbsp;<br>";

	$query = "SELECT title,threadid,lastpost FROM thread WHERE lastpost > '$ts1' AND lastpost < '$ts2' AND forumid='$forumID' ORDER BY dateline ASC";
	
	$data = $db->select($query);

	if(!empty($data)) {

		while ( list ( $key,$thread ) = each ($data)) {

			echo spacer(3)."<a href=\"$baseURL/$forumID/$year/$month/$week/$thread[threadid]\">$thread[title]</a><br>";
			//echo spacer(3)."<a href=\"http://www.overgrow.com/edge/showthread.php?threadid=$thread[threadid]\">$thread[title]</a><br>";
		}
	} else {
		echo spacer(3)."No posts, please <a href=\"".getenv(HTTP_REFERER)."\">go back</a>.";
	}

}

function spacer($n) {

	global $spacer;

	while($n!=0){
		$return.=$spacer;
		$n--;
	}
	return $return;
}

function removeZero($n) {

	preg_replace ("/./","",$n);
	return $n;
}


function weekList() {

	global $db,$baseURL,$forumID,$firstYear,$currentYear,$currentMonth,$firstMonth;

	$months = array("blank","January","February","March","April","May","June","July","August","September","October","November","December");

	echo "<a href=\"$baseURL/index.php\">Archives</a><br>";

	$query = "SELECT title FROM forum WHERE forumid='$forumID'";

	$data = $db->select($query);

	if(!empty($data)) {

		while ( list ( $key,$forum ) = each ($data)) {

			echo spacer(1)."$forum[title]<br>";
		}
	}

	for($y=$firstYear;$y<=$currentYear;$y++) {

		echo spacer(2)."$y<br>&nbsp;<br>";

		if($y==$firstYear)
			$fm=$firstMonth;
		else
			$fm=1;

		if($y==$currentYear)
			$lm=($currentMonth + 1);
		else
			$lm=13;
			

		for($m=$fm;$m<$lm;$m++){
			
			echo spacer(3).$months[$m]."<br>";

			for($w=1;$w<5;$w++) {

				if (strlen($m) < 2)
					$mo="0".$m;
				else
					$mo=$m;

				echo spacer(4)."<a href=\"$baseURL/$forumID/$y/$mo/$w\">week $w</a><br>";
			}

			echo "&nbsp;<br>";
		}
	}

}


function forumList() {

	global $db,$baseURL,$privateForums;

	echo "Archives<br>";

	$privateForums=preg_replace("/^\|/","",$privateForums);
	$privateForums=preg_replace("/\|$/","",$privateForums);

	$pfs = explode("|",$privateForums);
	$whereclause="";
	$wherecounter=0;


	while($pf=array_shift($pfs)) {
		if(ereg("[0-9]",$pf)) {

			if($wherecounter==0) {
				$whereclause= " WHERE forumid!='$pf'";
				$wherecounter++;
			} else {
				$whereclause .= " AND forumid!='$pf'";
			}
		}
	 }
	
	

	$query = "SELECT title,forumid FROM forum$whereclause ORDER BY forumid ASC";

	$data = $db->select($query);

	if(!empty($data)) {

		while ( list ( $key,$forum ) = each ($data)) {

			echo spacer(1)."<a href=\"$baseURL/$forum[forumid]\">$forum[title]</a><br>";
		}
	}	
}


function htmlHeader() {

	global $keywords,$description,$pagetitle;

?>

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML>
<HEAD>
<TITLE> <?=$pagetitle?> </TITLE>
<META NAME="Generator" CONTENT="EditPlus">
<META NAME="Author" CONTENT="">
<META NAME="Keywords" CONTENT="<?=$keywords?>">
<META NAME="Description" CONTENT="<?=$description?>">

<style type="text/css">

BODY {font: 10pt verdana,arial,sans-serif;}
TD {font: 10pt verdana,arial,sans-serif;}

</style>

</HEAD>

<BODY bgcolor="#ffffff">

<?
}

?>
Thanks Steven
Reply With Quote
Reply


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 05:53 AM.


Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2024, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.06183 seconds
  • Memory Usage 2,353KB
  • Queries Executed 25 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (1)bbcode_code
  • (3)bbcode_php
  • (4)bbcode_quote
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)modsystem_post
  • (1)navbar
  • (6)navbar_link
  • (120)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (4)pagenav_pagelink
  • (1)pagenav_pagelinkrel
  • (11)post_thanks_box
  • (11)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (11)post_thanks_postbit_info
  • (10)postbit
  • (10)postbit_onlinestatus
  • (11)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 

Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • fetch_musername
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • pagenav_page
  • pagenav_complete
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete