vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vBulletin 2.x Beta Releases (https://vborg.vbsupport.ru/forumdisplay.php?f=5)
-   -   Have Googlebot Crawl EVERYTHING (theoretically) (https://vborg.vbsupport.ru/showthread.php?t=50805)

zajako 04-14-2003 06:03 PM

i will install this when i get to my home machine !!

thanks filburt :]

Ryzer 06-01-2003 10:07 AM

THNX m8! :)

PixelFx 06-02-2003 09:25 AM

looks great, however I'm wondering how well this would work with robots.txt for sections of a site you don't want index'd ?

Skyline_GT 06-04-2003 01:43 AM

Nice hack
installed:D

gmarik 06-05-2003 02:07 PM

Filburt, how can I hide sessions forever?
Is there a hack to write the sessions in cookies and not show them for user at all?

Zach 06-06-2003 06:31 AM

Quote:

Yesterday at 08:07 AM gmarik said this in Post #35
Filburt, how can I hide sessions forever?
Is there a hack to write the sessions in cookies and not show them for user at all?


That would be pointless - you are trying to keep track of non cookied users.

Zach 06-06-2003 06:43 AM

I dunno, but I have the session hash in my urls and google crawls them, I have not got the results I hope to get yet, but I went from like a hundred to 1000 or so.

I am using something else I threw together to help google along but I know for sure that the hash is still there.

Zach 06-06-2003 06:52 AM

This is rather weird, my stats program stopped up dating a couple of of mornings ago - guess it got sick of scooter.

But this is the basically four days and four hours I guess of stats on search engines. Hum, now off to figure out whats up with my stats, or server time or what.

Scooter (AltaVista) 74909 4.53 GB 04 Jun 2003 - 22:57
Googlebot (Google) 10091 591.56 MB 04 Jun 2003 - 19:54
Inktomi Slurp 5833 320.94 MB 05 Jun 2003 - 04:07
Fast-Webcrawler (AllTheWeb) 3777 167.30 MB 05 Jun 2003 - 04:07
Alexa (IA Archiver) 1076 12.58 MB 05 Jun 2003 - 03:49
IBM_Planetwide 615 1.15 MB 04 Jun 2003 - 19:33
Road Runner: The ImageScape Robot 366 955.57 KB 04 Jun 2003 - 15:14
LinkWalker 137 9.79 MB 03 Jun 2003 - 13:01
SurveyBot 123 3.09 MB 02 Jun 2003 - 06:29
MSIECrawler 81 307.02 KB 02 Jun 2003 - 11:39
Pioneer 59 184.58 KB 01 Jun 2003 - 08:31
Unknown robot (identified by 'crawl') 54 1.67 MB 04 Jun 2003 - 15:54
InternetSeer 9 0 05 Jun 2003 - 02:08
arks 8 63.65 KB 04 Jun 2003 - 19:36
BaiDuSpider 7 289.54 KB 04 Jun 2003 - 05:12
WISENutbot (Looksmart) 6 108.84 KB 03 Jun 2003 - 23:19
Calif 6 25.09 KB 02 Jun 2003 - 16:11
WebClipping.com 3 23.83 KB 03 Jun 2003 - 11:24
Unknown robot (identified by 'spider') 2 99.97 KB 03 Jun 2003 - 15:08
ZealBot 2 30.25 KB 03 Jun 2003 - 06:07
Wget 1 49.42 KB 03 Jun 2003 - 16:56

I have no idea what got up scooters ass. I have never seen anything like it. He hit me for ten gigs the last week of May

dunno, but he can do what he please as long as I go over there and find my self number one for terms like

fantasy sports
sports forums

Zach 06-07-2003 02:33 PM

You know I can be so stupid at times, or I spend to much time in front of the computer )

Never considered that my computer clock/calendar would be off by two days :)

S.Shady 06-10-2003 01:26 AM

googlebot is on my forums now :) 16 mins on index.php :-\


how long does it take for the site results to be on google ?


All times are GMT. The time now is 07:27 AM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.02043 seconds
  • Memory Usage 1,735KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (4)pagenav_pagelink
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (10)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete