How do I keep google from indexing certain areas of forum?

[iamboard]

How do I keep google from indexing meaningless pages such as http://www.mysite.com/forum/calendar...ay=2003-10-16?

Any other good references that on what google does not really need to be indexing and how to keep it from doing so?



Thanks

[RichieBoy67]

robots.txt file..... do a google on it...

[iamboard]

Which areas of vbulletin are good to exclude in the robots.txt?

[Lynne]

Do a search in the thread titles and you will come up with a few. Here's mine (minus some site specific pages):

User-agent: *
Crawl-delay: 40
Disallow: /forums/announcement.php
Disallow: /forums/attachment.php
Disallow: /forums/calendar.php
Disallow: /forums/editpost.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/misc.php
Disallow: /forums/member.php
Disallow: /forums/memberlist.php
Disallow: /forums/moderator.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/sendtofriend.php
Disallow: /forums/showpost.php
Disallow: /forums/subscription.php
Disallow: /forums/subscriptions.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/images/

[SpartanSpence]

what is the crawl delay for??

[Lynne]

I have the crawl delay because otherwise yahoo slurp! spiders will take over my site.

[subzero06]

<a href="http://www.outfront.net/tutorials_02/adv_tech/robots.htm" target="_blank">http://www.outfront.net/tutorials_02...ech/robots.htm</a>