The Arcive of vBulletin Modifications Site.

Have Googlebot Crawl EVERYTHING (theoretically) · **Released**: 03-24-2003

Googlebot supposedly won't crawl links that have sessionhashes (32 character hex strings) in the URLs. So this hack detects when Googlebot is browsing your forum, and if it is, deletes the sessionhash. So, theoretically, Googlebot will be able to crawl every single guest-accessible link on your board. All of them. This means every thread, every forum, every post, even useless stuff like the memberlist and the FAQ.

Simple to install (1 step). This won't get instant gratification because Googlebot will only reindex your site on its next crawl.

Intex · #2 03-25-2003, 05:00 PM

Another great hack filburt1. Not sure I need it crawling my internal site, but nevertheless very cool.

colicab-d · #3 03-25-2003, 05:22 PM

lol so thats why u wanted the sessions code

filburt1 · #4 03-25-2003, 05:38 PM

No it's not...

VampireMan · #5 03-25-2003, 05:54 PM

simple but very effective hack , thanks :-)

Xanthine · #6 03-25-2003, 06:02 PM

Sounds good thanks Filburt

Overgrow · #7 03-25-2003, 06:35 PM

Excellent

One note: This may not allow Google to see all of your posts (theoretically). If you have old ones that would not naturally show up on forumdisplay, Google will not use the select box to see all of the old posts.

Elegantly simple though and makes perfect sense.

[high]* Overgrow clicks install[/high]

Davey · #8 03-25-2003, 06:42 PM

Filburt how did you find that special part of code through all that 'mess'?

.
Thanks.
*Installs*.

Dave.

Webdork · #9 03-25-2003, 08:31 PM

On my site I require registration for users to view threads. Is there a way to allow Google to become a "registered" user so it can crawl all threads still?

filburt1 · #10 03-25-2003, 08:32 PM

Yes but smart users can get around it...

edit: actually I thought of a way to fix that, too. Expect that hack eventually.
note to self: check first few IP bits and HTTP_USER_AGENT, update $bbuserinfo.