Have Googlebot Crawl EVERYTHING (theoretically)
 

Googlebot supposedly won't crawl links that have sessionhashes (32 character hex strings) in the URLs. So this hack detects when Googlebot is browsing your forum, and if it is, deletes the sessionhash. So, theoretically, Googlebot will be able to crawl every single guest-accessible link on your board. All of them. This means every thread, every forum, every post, even useless stuff like the memberlist and the FAQ.

Simple to install (1 step). This won't get instant gratification because Googlebot will only reindex your site on its next crawl.

Another great hack filburt1. Not sure I need it crawling my internal site, but nevertheless very cool.

lol so thats why u wanted the sessions code

No it's not...

simple but very effective hack , thanks :-)

Sounds good thanks Filburt

Excellent :)


One note: This may not allow Google to see all of your posts (theoretically). If you have old ones that would not naturally show up on forumdisplay, Google will not use the select box to see all of the old posts.

Elegantly simple though and makes perfect sense.


[high]* Overgrow clicks install[/high]

Filburt how did you find that special part of code through all that 'mess'?
;):p:D:).
Thanks.
*Installs*.

Dave.

On my site I require registration for users to view threads. Is there a way to allow Google to become a "registered" user so it can crawl all threads still?

Yes but smart users can get around it...

edit: actually I thought of a way to fix that, too. Expect that hack eventually.
note to self: check first few IP bits and HTTP_USER_AGENT, update $bbuserinfo.

installed ;) nice one

off topic

how r u lot doing those funky sigs ? i beleive u started the trend filb !

I dont really mind if a smart user can get in without registering - my forums arent private I just like to know who is using them. But the true value is of course in having as many threads as possible in Google to attract new users.

Nice hack, and I'll be awaiting that suggestion by Webdork in post #9 as well ;)

[high]* Velocd clicks install[/high]

Forgot it, I can't stand looking at sessions.php after trying for an hour.

Hmm, theres a similar hack for this in Tech's vB Archive hack that also includes several searchbots, not just google.

This is one file edit, don't think you need to install that beast...

You dont need to install vB Archive for the sessionhash stripping, you can find it in this post:
https://vborg.vbsupport.ru/showthrea...706#post342706

Just add a small function to functions.php and edit global.php and it strips all sessionhash's from search bots.

Also nice hack though filburt.

this is not working for me. At the top of my home page I get...
if (strpos(strtolower($_SERVER['HTTP_USER_AGENT']), "googlebot") != false) { unset($session['sessionhash']); }

any suggestions?

Read the instructions again.

ummmm... how did you do that? lol

EDIT: found the hack ;)

so this will do everything that the VBarchive hack will do with one file edit?

Well, it has the same objective as vBArchive, it doesn't do everything it does.

i recall that only 1 of the many steps of VBArchive was to remove the sessionhash's, that is why i ask

How is this working for everyone? Any results? I have it installed and saw a crawler today, but it didnt stay long....

Googlebot is swarming my forums right now so when the index is updated we should know if this works.

I have filbert1's template mod installed; therefore Google has already been crawling a lot of stuff. However, it never picked up the calendar or faq before. I put this hack in, and now my calendar is indexed on Google.

Amy

and google is registering, once again.

So far im showing no significant increase in google links, unfortunately. I will give it more time :)

any movement on this? i would like to know how effective it is to gauge whether or not i should install it! thanks...

Well after installing this, my google links dropped by 100, from 696 to 595. I was getting better results from a simple sitemap.

Not sure why this is, but obviously it doesnt work? Will wait for comments before uninstalling, but will take a month to fix the results with google.

i will install this when i get to my home machine !!

thanks filburt :]

THNX m8! :)

looks great, however I'm wondering how well this would work with robots.txt for sections of a site you don't want index'd ?

Nice hack
installed:D

Filburt, how can I hide sessions forever?
Is there a hack to write the sessions in cookies and not show them for user at all?

That would be pointless - you are trying to keep track of non cookied users.

I dunno, but I have the session hash in my urls and google crawls them, I have not got the results I hope to get yet, but I went from like a hundred to 1000 or so.

I am using something else I threw together to help google along but I know for sure that the hash is still there.

This is rather weird, my stats program stopped up dating a couple of of mornings ago - guess it got sick of scooter.

But this is the basically four days and four hours I guess of stats on search engines. Hum, now off to figure out whats up with my stats, or server time or what.

Scooter (AltaVista) 74909 4.53 GB 04 Jun 2003 - 22:57
Googlebot (Google) 10091 591.56 MB 04 Jun 2003 - 19:54
Inktomi Slurp 5833 320.94 MB 05 Jun 2003 - 04:07
Fast-Webcrawler (AllTheWeb) 3777 167.30 MB 05 Jun 2003 - 04:07
Alexa (IA Archiver) 1076 12.58 MB 05 Jun 2003 - 03:49
IBM_Planetwide 615 1.15 MB 04 Jun 2003 - 19:33
Road Runner: The ImageScape Robot 366 955.57 KB 04 Jun 2003 - 15:14
LinkWalker 137 9.79 MB 03 Jun 2003 - 13:01
SurveyBot 123 3.09 MB 02 Jun 2003 - 06:29
MSIECrawler 81 307.02 KB 02 Jun 2003 - 11:39
Pioneer 59 184.58 KB 01 Jun 2003 - 08:31
Unknown robot (identified by 'crawl') 54 1.67 MB 04 Jun 2003 - 15:54
InternetSeer 9 0 05 Jun 2003 - 02:08
arks 8 63.65 KB 04 Jun 2003 - 19:36
BaiDuSpider 7 289.54 KB 04 Jun 2003 - 05:12
WISENutbot (Looksmart) 6 108.84 KB 03 Jun 2003 - 23:19
Calif 6 25.09 KB 02 Jun 2003 - 16:11
WebClipping.com 3 23.83 KB 03 Jun 2003 - 11:24
Unknown robot (identified by 'spider') 2 99.97 KB 03 Jun 2003 - 15:08
ZealBot 2 30.25 KB 03 Jun 2003 - 06:07
Wget 1 49.42 KB 03 Jun 2003 - 16:56

I have no idea what got up scooters ass. I have never seen anything like it. He hit me for ten gigs the last week of May

dunno, but he can do what he please as long as I go over there and find my self number one for terms like

fantasy sports
sports forums

You know I can be so stupid at times, or I spend to much time in front of the computer )

Never considered that my computer clock/calendar would be off by two days :)

googlebot is on my forums now :) 16 mins on index.php :-\


how long does it take for the site results to be on google ?