Have Googlebot Crawl EVERYTHING (theoretically)

Googlebot supposedly won't crawl links that have sessionhashes (32 character hex strings) in the URLs. So this hack detects when Googlebot is browsing your forum, and if it is, deletes the sessionhash. So, theoretically, Googlebot will be able to crawl every single guest-accessible link on your board. All of them. This means every thread, every forum, every post, even useless stuff like the memberlist and the FAQ.

Simple to install (1 step). This won't get instant gratification because Googlebot will only reindex your site on its next crawl.

[Zach]

Yesterday at 07:26 PM S.Shady said this in Post #40
googlebot is on my forums now 16 mins on index.php :-\


how long does it take for the site results to be on google ?

Well, do not hold your breath waiting

IT can takes weeks to months, and will take months to start getting the good results. That also works in reverse, I have pages in google and others that have not existed for two years.

[S.Shady]

hmm damn so its not auto. ok then ill forget all about it then check again in about a month. Thanks now i wont be wasting time by checking for awhile. i can finally clean house

[GameCrash]

/me clicks install

[I, Brian]

I know this hack works because I joined filbert's design forum in it's early days, then lost the link in a re-format of my harddrive - a few months later I found a backlink from his forums.

He's shown by example how this hack can work. And google indexing of forums is precisely what sent me looking at vBulletin. Forum-Forum also had a big part to play in that when I asw how well they were indexed by Google.

Just starting up vB 2.3 and will implement filburt's hack as soon as I'm sure everything is running smoothly.

[Erwin]

vB3 implements this already automatically. Sessionhash is automatically disabled for Google, Inktomi etc.

[xs1]

i dont sessions.php file any help ? do i have to edit the one in admin folder ?

[clamcrusher]

will this allow googlebot to read forums you have set to private, like for example a forum i made that only mods and admins can view? what is talked about in there i dont want shared.

also i heard a rumor that it might be able to read pm's too?

[NTLDR]

will this allow googlebot to read forums you have set to private,

No

also i heard a rumor that it might be able to read pm's too?

and No All this does is remove the sessionhash in the URLs when google is crawling.

[BlueHawk]

Nice hack, so many peeps need relevent info on problems they are having, hopefully our forums will now help

[Pikok]

03-25-03 at 11:31 PM Webdork said this in Post #9
On my site I require registration for users to view threads. Is there a way to allow Google to become a "registered" user so it can crawl all threads still?

This is what I did and tested it with a couple of different Web robots and it appears to be working okay. Already having inphinity's UserAgent Checking hack installed, I just wrote a bit of code to work along with it to set permissions for the robots.

1. Install inphinity's UserAgent Checking add-on for TECK's vBArchive. UserAgent Checking works idependently of vBArchive.
2. Ceate a usergroup for Web robots and setup the various permissions as desired.
3. In "root/global.php" find:
   PHP Code:

   $permissions=getpermissions(); 

   And Add Above That:
   PHP Code:

   // Search Engine Perms Start
$sep_checkagent = $bbuserinfo['useragent'];
$sep_robot = useragentcheck($sep_checkagent, 1);
if ($sep_robot) {
  $bbuserinfo[usergroupid] = 10; //Set Usergroup to Match the Robot Usergroup
  $bbuserinfo[username] = $sep_robot; //Don't Think This Line's Really Needed, But I Make Use of It
}
// Search Engine Perms End 


I guess you could say this would be an add-on for an add-on.. lol Let me know if it works for you.