vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vB3 General Discussions (https://vborg.vbsupport.ru/forumdisplay.php?f=111)
-   -   How do I keep google from indexing certain areas of forum? (https://vborg.vbsupport.ru/showthread.php?t=143557)

iamboard 03-30-2007 12:02 AM
How do I keep google from indexing certain areas of forum?
 
How do I keep google from indexing meaningless pages such as http://www.mysite.com/forum/calendar...ay=2003-10-16?

Any other good references that on what google does not really need to be indexing and how to keep it from doing so?



Thanks

RichieBoy67 03-30-2007 01:25 AM
robots.txt file..... do a google on it...

iamboard 03-30-2007 12:39 PM
Which areas of vbulletin are good to exclude in the robots.txt?

Lynne 03-30-2007 02:47 PM
Do a search in the thread titles and you will come up with a few. Here's mine (minus some site specific pages):
Code:
User-agent: *
Crawl-delay: 40
Disallow: /forums/announcement.php
Disallow: /forums/attachment.php
Disallow: /forums/calendar.php
Disallow: /forums/editpost.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/misc.php
Disallow: /forums/member.php
Disallow: /forums/memberlist.php
Disallow: /forums/moderator.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/sendtofriend.php
Disallow: /forums/showpost.php
Disallow: /forums/subscription.php
Disallow: /forums/subscriptions.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/images/

SpartanSpence 03-31-2007 05:55 PM
what is the crawl delay for??

Lynne 03-31-2007 08:15 PM
I have the crawl delay because otherwise yahoo slurp! spiders will take over my site.

subzero06 04-01-2007 01:46 AM
<a href="http://www.outfront.net/tutorials_02/adv_tech/robots.htm" target="_blank">http://www.outfront.net/tutorials_02...ech/robots.htm</a>


All times are GMT. The time now is 07:05 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01030 seconds
  • Memory Usage 1,720KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)bbcode_code_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (7)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 
Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete