Google and other spiders accessing threads, member profiles etc... Help!

[Sam Granger]

Hey guys!

I am hoping that someone here might know the answer to my question :).

I have a forum that is blocked off for the public. It's an invite only forum and I do not want people outside the forum seeing any posts. Guests are forced to register and to be approved before they can take part. So far so good. However, I just saw amazonaws viewing a thread as a "guest"... How on earth is this possible? And all my pages are indexed in google? How can I just allow Google to find the index where guests are forced to login or register and go nowhere beyond that? Thanks for reading and I hope someone might be able to help me out in the right direction. :)

[Quantnet]

Use robots.txt file to control spiders and what they can see.

[hpidriver]

robots.txt is simply a suggestion of what you want followed, and what you don't.

Some bots ignore it totally. .htaccess is the only real way to prevent them from hitting specific pages.

[Sam Granger]

But why on earth does vBulletin allow bots to act like members? It's ridiculous! Does anyone have an htaccess/robots.txt example? vBulletin, if this is the only way to control robots not acting like members, you have seriously disappointed me :(

[Lynne]

They don't. If you look closely at the list, you will see a Stop sign next to the thread title.

[hpidriver]

#ALL BOTS
User-agent: *

Disallow: /forums/subscriptions/
Disallow: /forums/customavatars/
Disallow: /forums/customprofilepics/
Disallow: /forums/announcement.php
Disallow: /forums/attachment.php
Disallow: /forums/calendar.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/faq.php
Disallow: /forums/frm_attach
Disallow: /forums/image.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/memberlist.php
Disallow: /forums/members/
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/payment_gateway.php
Disallow: /forums/payments.php
Disallow: /forums/postings.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/subscription.php
Disallow: /forums/usercp.php
Disallow: /forums/threadrate.php
Disallow: /forums/usernote.php

Sitemap: http://www.yourdomain.com/sitemap_index.xml.gz

---------------------