Have Googlebot Crawl EVERYTHING (theoretically)

Googlebot supposedly won't crawl links that have sessionhashes (32 character hex strings) in the URLs. So this hack detects when Googlebot is browsing your forum, and if it is, deletes the sessionhash. So, theoretically, Googlebot will be able to crawl every single guest-accessible link on your board. All of them. This means every thread, every forum, every post, even useless stuff like the memberlist and the FAQ.

Simple to install (1 step). This won't get instant gratification because Googlebot will only reindex your site on its next crawl.

[xs1]

i dont sessions.php file any help ? do i have to edit the one in admin folder ?

[clamcrusher]

will this allow googlebot to read forums you have set to private, like for example a forum i made that only mods and admins can view? what is talked about in there i dont want shared.

also i heard a rumor that it might be able to read pm's too?

[NTLDR]

will this allow googlebot to read forums you have set to private,

No

also i heard a rumor that it might be able to read pm's too?

and No All this does is remove the sessionhash in the URLs when google is crawling.

[BlueHawk]

Nice hack, so many peeps need relevent info on problems they are having, hopefully our forums will now help

[Pikok]

03-25-03 at 11:31 PM Webdork said this in Post #9
On my site I require registration for users to view threads. Is there a way to allow Google to become a "registered" user so it can crawl all threads still?

This is what I did and tested it with a couple of different Web robots and it appears to be working okay. Already having inphinity's UserAgent Checking hack installed, I just wrote a bit of code to work along with it to set permissions for the robots.

1. Install inphinity's UserAgent Checking add-on for TECK's vBArchive. UserAgent Checking works idependently of vBArchive.
2. Ceate a usergroup for Web robots and setup the various permissions as desired.
3. In "root/global.php" find:
   PHP Code:

   $permissions=getpermissions(); 

   And Add Above That:
   PHP Code:

   // Search Engine Perms Start
$sep_checkagent = $bbuserinfo['useragent'];
$sep_robot = useragentcheck($sep_checkagent, 1);
if ($sep_robot) {
  $bbuserinfo[usergroupid] = 10; //Set Usergroup to Match the Robot Usergroup
  $bbuserinfo[username] = $sep_robot; //Don't Think This Line's Really Needed, But I Make Use of It
}
// Search Engine Perms End 


I guess you could say this would be an add-on for an add-on.. lol Let me know if it works for you.

[Salazar]

Thanks filburt1!

[high]* Salazar clicks install...[/high]

[Khashyar]

Thank you for the hack....

I just installed the EASY-to-install hack in my forums, and I will let you all know what happens.

Has anyone noticed an increase on their website's search engine listings after installing the hack?

Thank again, filburt

Khashyar
www.russiameetingplace.com/forums

[Khashyar]

filburt (and others...)

Would it be useful to also specify other search engines in a similar hack in the session.php file?

Thanks,

Khashyar

[Matrixgl]

Its not working....I did change admin/sessions.php about 10 days ago, still no results :'(

[dethfire]

hmmm doesn't seem to work, I see googlebot is look at this page /archive/topic/6075-1.html?

shouldn't the "s" string be unset?

[MaDCaT75]

Great hack.

[rikku3978]

I will use this, the more hits the better, right? :P

[CF Web]

*bump*

Anyone else with results on the effectiveness of this hack?

[zeeko1212]

very effective hack , thanks !

[arsec]

i dont have the sessions.php file, is that normal? im downloading it now